![]() |
Project ORB: A superb predictive scouting system!
Okay, so I've been waiting and waiting and waiting to be able to tell the FIRST community about this, and I'm beyond excited to share tonight, and even though it's not quite public yet, I'm getting this out here so people can start asking questions and making feature suggestions for this week.
I'm Sam Weaver, Lead Programmer for Team 4534: The Wired Wizards (archimedes division!) Early on in the season, our team's leadership sat down and discussed scouting this year, and determined that we would not be able to dedicate a full time scouting staff at events this year. Programming was tasked to solve this, and we came up with an idea that we think redefines scouting for FIRST Stronghold. Originating early on in the season under the codename "Project Magic Wand," the name later changed to Project ORB, to better reflect what it does. What does it do, you may ask? Project Orb is an automatic predictive scouting system, that uses a combination of Neural Networks and statistical analysis that makes predictions about teams capabilities, specifically defense capabilities, high and low goal shooting abilities, climbing ability, and challenge consistency, and match predictions statistically more likely to be accurate than OPR. In essence, without entering in or providing any data, any user can view the capabilities of a robot to cross defenses, how many goals they can score in a match, and whether they can climb or not, completely automatically, for all 3144 active teams in the 2016 FRC Season. Cool, right? How does it work? Project ORB uses TheBlueAlliance's API (thanks TBA!) to view and analyze all matcches a team has competed in over the entire season. This match data includes values about defense crossings, challenges and climbs, and high and low goal scores. Unfortunately, the data that the Field Management System at events provides to TheBlueAlliance is not spefici to which team completed which action, making the data along not very valuable. Here's where our friends The Neural Networks come in! In our training process, we create a Feed Forward Neural Network for each and every team, and train it on their match data, with the goal to eliminate noise from their random alliance partnerships during qualifications. The results, are predictions on how many crossings of a defense a robot can make (ranging from zero to two.) This was the first step in Project Magic Wand, and we then adapted it to be more accurate, and adapted it for goal scoring and climbing and challenges. How can I use it? We did a beta test with all the teams at the 2016 NC District Championship (thanks guys o/ ) within our scouting app platform, and have since moved the data out to a standalone site, available at http://orb.scoutfrc.com/. Our programming team is still finishing up touches on the front end, so it is on our coming soon page at the moment, but we aim to have the site up and running on Thursday. What are the significance of these numbers? How accurate are these results? Inside the system, you will receive proficiency percentages for each defense on a team's page. These percentages range from 0-100, and are typically mapped to the raw output of our networks, which range from zero to two. Unfortunately, due to how we gather the data, and trends in the game, some results are less accurate than others. Specifically, the Low Bar defense, for example, is almost never below a value of 1.5, due to the fact that there is a low bar robot in almost every match. As a result, the percentage for this defense will be tweaked to reflect a range that is more useful, such as 1.5-2. In contrast, you receive actual numbers for low and high goal shooting, which represent how many goals the system predicts they can score during a match.These numbers are broken up into auto and teleop values, which can allow you to predict a team's auto routine. Strengths we've found in this system come in the ability to provide data in bulk, in supplement, or before your scouting team can get it. In addition, our system gives proficiency percentages of defenses, so while a team might report that they can complete both defenses in a defense group, our system can report which of the two they are weaker at, trumping what would typically be a boolean flag. Please, please, ask questions. We want to get this system to the best possible state for this week's competition. We rushed to put this post out tonight so that everyone could get the information as soon as possible, we can provide more data as we get more chances to figure out what people need to know, and as we keep tweaking our system. Last but not least, I'd like to give special thanks to two of my programmers, Tom and Danny, for their extensive work on this project, and the rest of the programming team for their superb work this season. I wish all teams the best of luck, and we'll see you at the competition! Sam Weaver, Wired Wizards 4534 |
Re: Project ORB: A superb predictive scouting system!
This is amazing. Thanks for the contribution I'm going to be taking a look in depth next thing tomorrow.
Good luck in Archimedes btw! |
Re: Project ORB: A superb predictive scouting system!
It's like OPR calculation on steroids, can't wait to check it our tomorrow!
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
How accurate were your results at NC DCMP? Did you compare predictions with actual (eg. ORB provides a team should get 3 high goals, but in reality get something different)?
|
Re: Project ORB: A superb predictive scouting system!
Really interested to take a look at the system, and it sounds super exciting! I've actually been working on training a Deep Learning system for FRC the last couple of years using some stuff I've been using in research here at Berkeley to do something quite similar. As of right now, all I've been doing is just dumping data from TBA and tweaking the training sets an a few external heuristics. I just never get a chance to do something real time. It did really really well for District Championships, but St. Louis is always another beast of its own. Feel free to shoot me a PM after Champs or find me in our pit if you're interested at all for using that next year. I've been trying to integrate my rig with someone with a bit of UX skill because I definitely have none!
EDIT: I also just remembered I'll be working on really similar core tech @ Google this summer. |
Re: Project ORB: A superb predictive scouting system!
weaverSam8,
Sounds like a great idea and a great use of computer-thinking to figure things out. But I think I see a gap? Just because the statistics show that a robot 'hasn't' done something doesn't mean that it can't. For instance, in it's whole life, our robot has only made like two low goals. There's really seldom a reason to even try--we can make high goals at a high percentage. We only did those two in eliminations to beat defenders. Same goes for low bar and rough terrain. Yes we can low-bar, but we have 10-inch wheels and good manipulators so we can cross all the defenses without much trouble. Almost always there are robots on our alliance that would prefer the low bar or the RT, so we 'give' it to them. But I can see how the statistics may still show that my team scored on lowbar or RT, because likely someone on the alliance did, so maybe we'd get the points? I don't see how we'd ever be given points for being able to low-goal shoot. How will your neural network evaluate a robot that just hasn't done these? |
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
This sounds interesting, but I'm cautiously skeptical about the use of neural nets and their benefits in this application compared to a statistical model. Nonetheless, I'm excited to see the results!
|
Re: Project ORB: A superb predictive scouting system!
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.
|
Re: Project ORB: A superb predictive scouting system!
I have thought that cool things could be done with regards to scouting+machine learning. It is cool to see someone exploring the's tools. I am looking forward to seeing more details about your system.
I don't think it is a stretch to say a Neural Network can learn a model of robot performance that is much better than OPR, but you have not shown that your Neural Networks perform better or not. Do you have any data of how well you predictions perform? Generally you perform a train-test split to see how well you can predict a robot's contribution, and compare it to the robots actual contribution and see how well it matches. I might have a few more thoughts later, but I am phone posting and don't want to type out too much more right now. Regardless of how effective this tool is, it is still a cool project to learn about neural networks. |
Re: Project ORB: A superb predictive scouting system!
Quote:
Sometimes it works out to trust another team with it, and other times it doesn't. |
Re: Project ORB: A superb predictive scouting system!
Quote:
I regret not being able to respond to all the questions, our team has had a very busy day today. Please, in the immediate, accept this: Update on ORB: We've decided to release tonight ORB in it's current working state, even though it has a handful of bugs. We hope to squash these over this week, but we wanted to make sure teams could use the data so that everyone is on the same playing field. ORB Beta will be available around 11:00 PM CST at http://orb.scoutfrc.com/. Please report bugs and feedback in this thread, we'll try to be available on CD to answer more questions tomorrow, but if you're at championships, feel free to stop by our pit, we're 4534 in Archimedes, Row U. Until we provide some answers to some of your questions, we hope the results you'll see tomorrow speak for themselves. Good luck folks! |
Re: Project ORB: A superb predictive scouting system!
Looks like Curie isn't showing up for me. Can anyone confirm?
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
The website is showing the "loading teams..." message for an awfully long time for me (it never loads the teams :(). Should I be worried?
EDIT: Carver loads fine, for some reason, although there is still a delay in opening. DOUBLEEDIT: Also, what do the numbers mean? Is it all relative measurement? |
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Looks like it is working now. Great job getting it up. Thanks for the work and dedication this took and allowing it to be usable for everyone.
|
Re: Project ORB: A superb predictive scouting system!
This is incredible. I love the fact that I can still pull up data for teams that are not at champs. The data looks pretty darn accurate. I have two questions:
1) Will these be updating as new matches are played during champs? 2) Could you release a top 50/100/3000? I'd be interested to see how your program ranks the robots, and compare it to OPR and my personal favorites. I could (and might) compile this myself, but it'd take a while, and you might have some way of doing it much more quickly. |
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Good morning everybody!
Quote:
Match Scores For match scores, our system takes the highest skill level of each individual defense and assumes that is the score of the robot crossing the defense, then it uses that for purposes of scoring. Then, it calculates in low and high goal shots, and then calculates the scale/challenge bonus. Quote:
I can do a data dump of whatever teams you like, including the ones at the MAR Championship, but right now, data in the system will be skewed by the fact that the matches at that championship became part of the team's score, which means it could be less truly accurate. We might have the option to release some data from earlier in the season, but we'd have to dedicate training resources for that, which may be one of our lower priorities at the moment. We'll see what we can do. Coming next:
|
Re: Project ORB: A superb predictive scouting system!
How is the data represented for the neural network? I was thinking about this the other day, and wasn't too sure how you'd account for individual teams without essentially combining the teams' stats into one "super team" (for a lack of a better phrase).
|
Re: Project ORB: A superb predictive scouting system!
I haven't looked at how accurate ORBs predictions have been, but I noticed that it is consistently overestimating scores by a large margin. Any insight as to why this might be occurring?
|
Re: Project ORB: A superb predictive scouting system!
its down
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Howdy!! I'm Tom, Wired Wizards' Alpha Team leader (Robot Code), and I as well have been doing the vast majority of the back end for ORB.
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Source code?
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Hope everybody's been having a great day! We have found that there are some issues with retraining teams as matches occur, so we've turned off that feature in order to ensure that the web service remains online. We'll train tonight for all teams at Championships to make sure the data tomorrow is fresh based on how teams played today.
Quote:
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Quote:
Code:
http://orb.scoutfrc.com/#/a/event/<EVENT CODE> |
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Quote:
Sorry, I may have not given enough information there. Those predictions were based on the data we had up to that point, and not after, with the data we have now. If the bots in eliminations then had the capabilities they do now, it might have turned out different. We were also using a different ranking system at the time, our new system has been adjusted, but may not apply to some of our older results. |
Re: Project ORB: A superb predictive scouting system!
Quote:
Edit: My programming team lead (Sam; weaversam8) has informed to let you know that he is tired and delirious and that you should only pay attention to this response to your question, not his. |
ORB back-end API public release!
As ORB's back-end nears completion, we are releasing access to it! This is what the currently accessible front-end uses to get the info. There are just a small few of endpoints that are not implemented, and as such will not be listed, but in the future they will be added. Otherwise, everything else is currently functioning as intended! (A side note, as mentioned above, scoring for anything that is not a qualifying match is currently wrong as time has not been available to carve the algorithm for the different scoring playoffs uses, but before World's playoffs start we will have it ready.)
http://orb.scoutfrc.io/ ORB API Endpoints: GET /list/<eventcode> — Returns a list of teams at event identified by eventcode. GET /team/<number> — Returns a basic JSON object about that team. GET /team/<number>/defense — Returns the entire defense skill lineup for that team. GET /team/<number>/defense/:defensenumber — Returns the defense skill for that team on defense X. GET /team/<number>/goals/high — Gets a teams skill at a high goal in teleop/auto. GET /team/<number>/goals/low — Gets a team's skill at a low goal in teleop/auto. GET /team/<number>/scale/ — Gets a team's skill as a percent decimal (0 to 1) at scaling. GET /team/<number>/challenge/ — Gets a team's skill as a percent decimal (0 to 1) at challenging. GET /team/<number>/score — Returns the team’s score for rankings. GET /work/match/<eventcode>/<matchidentifier>/ — Calculates result of a match by comparing teams’ scores, and ranking alliance scores, returns two values, first is red's score, second is blue's score. GET /work/defense/<eventcode>/<matchidentifier>/ — Calculates the optimal defense selection for both alliances, returns array with arrays for red and blue, arrays in those: first array is defenses in defense categories they are best at, second is array for defenses in defense categories they are worst at. Edit: event code and match identifier formats are the same as TBA's. |
Re: Project ORB: A superb predictive scouting system!
Good morning folks, and congratulations to all the teams who were selected for eliminations matches (we were not.)
As an update, and a testament to ORB's match prediction abilities, we are going to try and run some statistics to predict advancement of alliances in each division here at champs. I'll keep you up to date as much as we can. |
Re: Project ORB: A superb predictive scouting system!
Proud to announce that quarterfinals predictions are available in ORB as of now. It's important to realize that each match in a series (i.e: Q1M1,Q1M2,Q1M3) are all predicted with the same score based on how our system works. This is the best we can do. Therefore, if you look at the winning score for each Quarterfinal Series, you can see the predicted advancing alliance by looking at the winner.
Here are quick links in case you don't want to navigate the system (scroll to the bottom): Archimedes Carson Carver Curie Galileo Hopper Newton Tesla |
Re: Project ORB: A superb predictive scouting system!
After this weekend, I'd like to thank the 118+ teams that used Project ORB. This was a very useful tool for us, and an excellent learning experience. We hope you found it useful!
We plan on continuing Project Orb in the fall, so please feel free to follow up with us as next year's season approaches. I'm sure we all need some time to recover :) . I'm glad we've all had a great time sieging the Stronghold- we'll see you next year!! <3 Wired Wizards |
Re: Project ORB: A superb predictive scouting system!
Can you post how well your predictions worked against actual results?
|
Re: Project ORB: A superb predictive scouting system!
Quote:
|
Re: Project ORB: A superb predictive scouting system!
Would it be possible to prepare ORB numbers for IRI? This system is really cool, and with IRI looming, I was wondering if we could see ORB output for the IRI teams?
|
Hi Wired Wizards, I'm from team 587 the hedgehogs- are you planning on publishing data during events in real time or are you guys planning to keep if for yourselves?
Sent from my iPhone using Tapatalk |
Re: Project ORB: A superb predictive scouting system!
Hey guys!
Unfortunately, our backend was taken down after the season ended to save costs. Otherwise we'd run the numbers, of course. Now, what I can do for you, if your team is really serious about doing analysis, is link you to the server source code. It's very messy and unclean, but we don't have programming members able to work on it currently. Going forward, we plan to maintain ORB, and yes, we plan on continuing to publish data live during events. I'll make a post in a few minutes with the source. |
Re: Project ORB: A superb predictive scouting system!
Good evening everyone!
As requested, we are publishing the source to the ORB backend. It is my regret to publish the source, as it is lacking documentation, and not clean, but I have faith in the FRC community, that if someone really wants to use it, they'll be able to, and we'll attempt to provide the best support for this legacy code that we can. Here's a snippet from what I posted in the repository readme: Quote:
https://github.com/4534-WiredWizards/orb-backend Have fun folks! |
Re: Project ORB: A superb predictive scouting system!
Quote:
|
| All times are GMT -5. The time now is 12:53. |
Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi