Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Championship Event (http://www.chiefdelphi.com/forums/forumdisplay.php?f=12)
-   -   Project ORB: A superb predictive scouting system! (http://www.chiefdelphi.com/forums/showthread.php?t=147817)

weaversam8 27-04-2016 01:14

Project ORB: A superb predictive scouting system!
 
Okay, so I've been waiting and waiting and waiting to be able to tell the FIRST community about this, and I'm beyond excited to share tonight, and even though it's not quite public yet, I'm getting this out here so people can start asking questions and making feature suggestions for this week.



I'm Sam Weaver, Lead Programmer for Team 4534: The Wired Wizards (archimedes division!) Early on in the season, our team's leadership sat down and discussed scouting this year, and determined that we would not be able to dedicate a full time scouting staff at events this year. Programming was tasked to solve this, and we came up with an idea that we think redefines scouting for FIRST Stronghold.



Originating early on in the season under the codename "Project Magic Wand," the name later changed to Project ORB, to better reflect what it does. What does it do, you may ask? Project Orb is an automatic predictive scouting system, that uses a combination of Neural Networks and statistical analysis that makes predictions about teams’ capabilities, specifically defense capabilities, high and low goal shooting abilities, climbing ability, and challenge consistency, and match predictions statistically more likely to be accurate than OPR. In essence, without entering in or providing any data, any user can view the capabilities of a robot to cross defenses, how many goals they can score in a match, and whether they can climb or not, completely automatically, for all 3144 active teams in the 2016 FRC Season. Cool, right?



How does it work?

Project ORB uses TheBlueAlliance's API (thanks TBA!) to view and analyze all matcches a team has competed in over the entire season. This match data includes values about defense crossings, challenges and climbs, and high and low goal scores. Unfortunately, the data that the Field Management System at events provides to TheBlueAlliance is not spefici to which team completed which action, making the data along not very valuable. Here's where our friends The Neural Networks come in! In our training process, we create a Feed Forward Neural Network for each and every team, and train it on their match data, with the goal to eliminate noise from their random alliance partnerships during qualifications. The results, are predictions on how many crossings of a defense a robot can make (ranging from zero to two.) This was the first step in Project Magic Wand, and we then adapted it to be more accurate, and adapted it for goal scoring and climbing and challenges.



How can I use it?

We did a beta test with all the teams at the 2016 NC District Championship (thanks guys o/ ) within our scouting app platform, and have since moved the data out to a standalone site, available at http://orb.scoutfrc.com/. Our programming team is still finishing up touches on the front end, so it is on our coming soon page at the moment, but we aim to have the site up and running on Thursday.



What are the significance of these numbers? How accurate are these results?

Inside the system, you will receive proficiency percentages for each defense on a team's page. These percentages range from 0-100, and are typically mapped to the raw output of our networks, which range from zero to two. Unfortunately, due to how we gather the data, and trends in the game, some results are less accurate than others. Specifically, the Low Bar defense, for example, is almost never below a value of 1.5, due to the fact that there is a low bar robot in almost every match. As a result, the percentage for this defense will be tweaked to reflect a range that is more useful, such as 1.5-2. In contrast, you receive actual numbers for low and high goal shooting, which represent how many goals the system predicts they can score during a match.These numbers are broken up into auto and teleop values, which can allow you to predict a team's auto routine. Strengths we've found in this system come in the ability to provide data in bulk, in supplement, or before your scouting team can get it. In addition, our system gives proficiency percentages of defenses, so while a team might report that they can complete both defenses in a defense group, our system can report which of the two they are weaker at, trumping what would typically be a boolean flag.



Please, please, ask questions. We want to get this system to the best possible state for this week's competition. We rushed to put this post out tonight so that everyone could get the information as soon as possible, we can provide more data as we get more chances to figure out what people need to know, and as we keep tweaking our system.



Last but not least, I'd like to give special thanks to two of my programmers, Tom and Danny, for their extensive work on this project, and the rest of the programming team for their superb work this season. I wish all teams the best of luck, and we'll see you at the competition!



Sam Weaver, Wired Wizards 4534

Gravity 27-04-2016 01:21

Re: Project ORB: A superb predictive scouting system!
 
This is amazing. Thanks for the contribution I'm going to be taking a look in depth next thing tomorrow.

Good luck in Archimedes btw!

asid61 27-04-2016 01:52

Re: Project ORB: A superb predictive scouting system!
 
It's like OPR calculation on steroids, can't wait to check it our tomorrow!

SoftwareBug2.0 27-04-2016 02:53

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1579656)
Project Orb is ... more likely to be accurate than OPR.

I eagerly await proof. I hope you're looking at something more than goodness of fit.

Dezion 27-04-2016 06:16

Re: Project ORB: A superb predictive scouting system!
 
How accurate were your results at NC DCMP? Did you compare predictions with actual (eg. ORB provides a team should get 3 high goals, but in reality get something different)?

araniaraniratul 27-04-2016 06:46

Re: Project ORB: A superb predictive scouting system!
 
Really interested to take a look at the system, and it sounds super exciting! I've actually been working on training a Deep Learning system for FRC the last couple of years using some stuff I've been using in research here at Berkeley to do something quite similar. As of right now, all I've been doing is just dumping data from TBA and tweaking the training sets an a few external heuristics. I just never get a chance to do something real time. It did really really well for District Championships, but St. Louis is always another beast of its own. Feel free to shoot me a PM after Champs or find me in our pit if you're interested at all for using that next year. I've been trying to integrate my rig with someone with a bit of UX skill because I definitely have none!

EDIT: I also just remembered I'll be working on really similar core tech @ Google this summer.

hrench 27-04-2016 09:24

Re: Project ORB: A superb predictive scouting system!
 
weaverSam8,

Sounds like a great idea and a great use of computer-thinking to figure things out.

But I think I see a gap? Just because the statistics show that a robot 'hasn't' done something doesn't mean that it can't.

For instance, in it's whole life, our robot has only made like two low goals. There's really seldom a reason to even try--we can make high goals at a high percentage. We only did those two in eliminations to beat defenders.

Same goes for low bar and rough terrain. Yes we can low-bar, but we have 10-inch wheels and good manipulators so we can cross all the defenses without much trouble. Almost always there are robots on our alliance that would prefer the low bar or the RT, so we 'give' it to them.

But I can see how the statistics may still show that my team scored on lowbar or RT, because likely someone on the alliance did, so maybe we'd get the points?

I don't see how we'd ever be given points for being able to low-goal shoot.

How will your neural network evaluate a robot that just hasn't done these?

Ether 27-04-2016 09:31

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1579656)
...match predictions statistically more likely to be accurate than OPR... Please, please, ask questions.

What additional data are you using that makes match predictions statistically more likely to be accurate than component OPR?



Eugene Fang 27-04-2016 09:48

Re: Project ORB: A superb predictive scouting system!
 
This sounds interesting, but I'm cautiously skeptical about the use of neural nets and their benefits in this application compared to a statistical model. Nonetheless, I'm excited to see the results!

Brian Maher 27-04-2016 11:33

Re: Project ORB: A superb predictive scouting system!
 
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.

tallen 27-04-2016 11:59

Re: Project ORB: A superb predictive scouting system!
 
I have thought that cool things could be done with regards to scouting+machine learning. It is cool to see someone exploring the's tools. I am looking forward to seeing more details about your system.

I don't think it is a stretch to say a Neural Network can learn a model of robot performance that is much better than OPR, but you have not shown that your Neural Networks perform better or not.

Do you have any data of how well you predictions perform? Generally you perform a train-test split to see how well you can predict a robot's contribution, and compare it to the robots actual contribution and see how well it matches.

I might have a few more thoughts later, but I am phone posting and don't want to type out too much more right now.

Regardless of how effective this tool is, it is still a cool project to learn about neural networks.

asid61 27-04-2016 11:59

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by hrench (Post 1579735)
weaverSam8,

Sounds like a great idea and a great use of computer-thinking to figure things out.

But I think I see a gap? Just because the statistics show that a robot 'hasn't' done something doesn't mean that it can't.

From a scouting/strategy perspective, for anything but the low bar I would not be comfortable trusting a team that says they can do something they have not demonstrated before. A team that has never done something, in my eyes, is almost equivalent to a robot that cannot do something, barring very rare circumstances.
Sometimes it works out to trust another team with it, and other times it doesn't.

weaversam8 27-04-2016 23:50

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Dezion (Post 1579686)
How accurate were your results at NC DCMP? Did you compare predictions with actual (eg. ORB provides a team should get 3 high goals, but in reality get something different)?

We don't have metrics on their goals predictions in specific on hand, we'd have to retrain the historical data and compare it to match data. What we can tell you is that our match prediction system (in a state less accurate than it is now,) was able to predict 6 out of 7 of the advancements in the finals, including predicting correctly lower seeded alliances triumphing over higher seeds, a victory even we humans didn't quite predict.

I regret not being able to respond to all the questions, our team has had a very busy day today. Please, in the immediate, accept this:

Update on ORB:
We've decided to release tonight ORB in it's current working state, even though it has a handful of bugs. We hope to squash these over this week, but we wanted to make sure teams could use the data so that everyone is on the same playing field.

ORB Beta will be available around 11:00 PM CST at http://orb.scoutfrc.com/. Please report bugs and feedback in this thread, we'll try to be available on CD to answer more questions tomorrow, but if you're at championships, feel free to stop by our pit, we're 4534 in Archimedes, Row U. Until we provide some answers to some of your questions, we hope the results you'll see tomorrow speak for themselves.

Good luck folks!

Kyler Hagler 28-04-2016 00:26

Re: Project ORB: A superb predictive scouting system!
 
Looks like Curie isn't showing up for me. Can anyone confirm?

weaversam8 28-04-2016 00:29

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Kyler Hagler (Post 1579932)
Looks like Curie isn't showing up for me. Can anyone confirm?

We noticed the error in the logs before you reported it- we're looking into it right now. It seems that we're missing data points for 3 out of 3144 teams, and one of them happens to be in Curie...

asid61 28-04-2016 00:32

Re: Project ORB: A superb predictive scouting system!
 
The website is showing the "loading teams..." message for an awfully long time for me (it never loads the teams :(). Should I be worried?

EDIT: Carver loads fine, for some reason, although there is still a delay in opening.
DOUBLEEDIT: Also, what do the numbers mean? Is it all relative measurement?

weaversam8 28-04-2016 00:34

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1579933)
We noticed the error in the logs before you reported it- we're looking into it right now. It seems that we're missing data points for 3 out of 3144 teams, and one of them happens to be in Curie...

We have discovered that we're missing data for Teams 4, 8, and 11 for scaling. We're going to inject artificial data for those teams (just for that single value) tonight, and tomorrow update it with the most accurate data. That's all we can do tonight.

Kyler Hagler 28-04-2016 00:43

Re: Project ORB: A superb predictive scouting system!
 
Looks like it is working now. Great job getting it up. Thanks for the work and dedication this took and allowing it to be usable for everyone.

microbuns 28-04-2016 01:23

Re: Project ORB: A superb predictive scouting system!
 
This is incredible. I love the fact that I can still pull up data for teams that are not at champs. The data looks pretty darn accurate. I have two questions:

1) Will these be updating as new matches are played during champs?
2) Could you release a top 50/100/3000? I'd be interested to see how your program ranks the robots, and compare it to OPR and my personal favorites. I could (and might) compile this myself, but it'd take a while, and you might have some way of doing it much more quickly.

GKrotkov 28-04-2016 06:22

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by BMSOTM (Post 1579760)
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.

I second this like nobody's business. I would love to compare ORB with 1712's scouting data for MARCMP.

weaversam8 28-04-2016 06:59

Re: Project ORB: A superb predictive scouting system!
 
Good morning everybody!

Quote:

Originally Posted by microbuns (Post 1579949)
This is incredible. I love the fact that I can still pull up data for teams that are not at champs. The data looks pretty darn accurate. I have two questions:

1) Will these be updating as new matches are played during champs?
2) Could you release a top 50/100/3000? I'd be interested to see how your program ranks the robots, and compare it to OPR and my personal favorites. I could (and might) compile this myself, but it'd take a while, and you might have some way of doing it much more quickly.

We do plan on updating based on matches being played, not sure if it's live this morning, but if it's not, it will be as soon as possible. Also, I'll take note of your request about the top lists, we'll see if we can do it. It might interest you to understand our match scoring system:

Match Scores
For match scores, our system takes the highest skill level of each individual defense and assumes that is the score of the robot crossing the defense, then it uses that for purposes of scoring. Then, it calculates in low and high goal shots, and then calculates the scale/challenge bonus.

Quote:

Originally Posted by BMSOTM (Post 1579760)
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.

In regard to releasing previous data:
I can do a data dump of whatever teams you like, including the ones at the MAR Championship, but right now, data in the system will be skewed by the fact that the matches at that championship became part of the team's score, which means it could be less truly accurate. We might have the option to release some data from earlier in the season, but we'd have to dedicate training resources for that, which may be one of our lower priorities at the moment. We'll see what we can do.

Coming next:
  • We're looking into building a separate match ranking system that only takes this event into account, like how OPR individually examines an event.
  • Top X teams list? (You can already kindof do this in a division by sorting)
  • Slackbot to interface with the ORB API
  • Recommended defenses for a match view
  • And more...

kinganu123 28-04-2016 09:53

Re: Project ORB: A superb predictive scouting system!
 
How is the data represented for the neural network? I was thinking about this the other day, and wasn't too sure how you'd account for individual teams without essentially combining the teams' stats into one "super team" (for a lack of a better phrase).

Eugene Fang 28-04-2016 10:30

Re: Project ORB: A superb predictive scouting system!
 
I haven't looked at how accurate ORBs predictions have been, but I noticed that it is consistently overestimating scores by a large margin. Any insight as to why this might be occurring?

CrazyMohawk 28-04-2016 10:38

Re: Project ORB: A superb predictive scouting system!
 
its down

danielrw 28-04-2016 10:41

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by CrazyMohawk (Post 1580044)
its down

Thanks, we're working on getting it back up.

TomAwezome 28-04-2016 10:44

Re: Project ORB: A superb predictive scouting system!
 
Howdy!! I'm Tom, Wired Wizards' Alpha Team leader (Robot Code), and I as well have been doing the vast majority of the back end for ORB.

Quote:

Originally Posted by kinganu123 (Post 1580024)
How is the data represented for the neural network? I was thinking about this the other day, and wasn't too sure how you'd account for individual teams without essentially combining the teams' stats into one "super team" (for a lack of a better phrase).

The data from the neural networks is saved into a database into tables for goals, defenses, challenge, and scale; goals has 4 values (autolow, autohigh, teleoplow, teleophigh), defenses has 9 values (0-8, 0:lowbar, 1:portcullis, 2:cdf, 3:moat, 4:ramparts, 5:drawbridge, 6:sallyport, 7:rockwall, 8:roughterrain), scale and challenge each have 1 value for their respective percentage. The goals values are representative of the quantity of goals, defenses representative of 0-2 crossings (as that is all TBA has per match), scale and challenge as mentioned are stored as a value between 0-1, a decimal percent. As each match is a combination of 3 teams per alliance, pulling the data gives you an idea of how they perform, but of course it is just a showing of how the alliance performs! The magic happens as the networks for each team analyze all of their different combinations, and finds the trends inside the dataset that would indicate the team it's training for. Good question!

danielrw 28-04-2016 11:05

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by danielrw (Post 1580045)
Thanks, we're working on getting it back up.

We restarted the server and everything seems to be working now.

marshall 28-04-2016 11:57

Re: Project ORB: A superb predictive scouting system!
 
Source code?

TomAwezome 28-04-2016 12:10

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Eugene Fang (Post 1580042)
I haven't looked at how accurate ORBs predictions have been, but I noticed that it is consistently overestimating scores by a large margin. Any insight as to why this might be occurring?

As far as I could tell much earlier in development, it was oddly over in estimation, but it seemed consistent by a certain multiplier (not impacting the predictions for winning, just showing bigger values.) Before, I had multiplied it by about .7310 to get the scores looking more like actual match scores. It is currently not doing that, since as mentioned it has no impact on prediction. The score value is a result of how it's summing it all, as it is trying to follow as close as possible the way that match scoring works in-game, but is not caused by any errors.

weaversam8 28-04-2016 13:50

Re: Project ORB: A superb predictive scouting system!
 
Hope everybody's been having a great day! We have found that there are some issues with retraining teams as matches occur, so we've turned off that feature in order to ensure that the web service remains online. We'll train tonight for all teams at Championships to make sure the data tomorrow is fresh based on how teams played today.

Quote:

Originally Posted by TomAwezome (Post 1580068)
As far as I could tell much earlier in development, it was oddly over in estimation, but it seemed consistent by a certain multiplier (not impacting the predictions for winning, just showing bigger values.) Before, I had multiplied it by about .7310 to get the scores looking more like actual match scores. It is currently not doing that, since as mentioned it has no impact on prediction. The score value is a result of how it's summing it all, as it is trying to follow as close as possible the way that match scoring works in-game, but is not caused by any errors.

We just did a tweak and they should be slightly more realistic, but we'll be looking into adjusting these values tonight to make them appear more like match scores.

Quote:

Originally Posted by marshall (Post 1580062)
Source code?

We plan on releasing the source code at some point, it's a bit fragmented at the moment and we'll need to clean it up.

microbuns 28-04-2016 13:57

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by GKrotkov (Post 1579970)
I second this like nobody's business. I would love to compare ORB with 1712's scouting data for MARCMP.

MARCMP. In order to get a specific event, go to the Blue Alliance, find the event you want, and navigate to its event page. In the URL, after the last slash, there will be the "event code" - something like 2016mrcmp (for MAR champs). Take this event code, and place it in this URL:

Code:

http://orb.scoutfrc.com/#/a/event/<EVENT CODE>

kinganu123 28-04-2016 15:26

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by TomAwezome (Post 1580047)
Howdy!! I'm Tom, Wired Wizards' Alpha Team leader (Robot Code), and I as well have been doing the vast majority of the back end for ORB.



The data from the neural networks is saved into a database into tables for goals, defenses, challenge, and scale; goals has 4 values (autolow, autohigh, teleoplow, teleophigh), defenses has 9 values (0-8, 0:lowbar, 1:portcullis, 2:cdf, 3:moat, 4:ramparts, 5:drawbridge, 6:sallyport, 7:rockwall, 8:roughterrain), scale and challenge each have 1 value for their respective percentage. The goals values are representative of the quantity of goals, defenses representative of 0-2 crossings (as that is all TBA has per match), scale and challenge as mentioned are stored as a value between 0-1, a decimal percent. As each match is a combination of 3 teams per alliance, pulling the data gives you an idea of how they perform, but of course it is just a showing of how the alliance performs! The magic happens as the networks for each team analyze all of their different combinations, and finds the trends inside the dataset that would indicate the team it's training for. Good question!

Ok, so my followup question is how are you combining the teams into an alliance? Do you use something like CCWM to weight each team into a single alliance?

TomAwezome 28-04-2016 16:00

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by kinganu123 (Post 1580094)
Ok, so my followup question is how are you combining the teams into an alliance? Do you use something like CCWM to weight each team into a single alliance?

The only data it uses is data it has pulled then trained. To compute how a combination of teams adds up as an alliance, it, for each defense, finds the max of the three teams' defense scores for that defense, then multiplies that by either 5 for lowbar and 2.5 for the other defenses. The same for goals, but with the proper point values for auto/teleop high/goal. And again, the same for scale and challenge, with the proper score multipliers. It then adds these all together and the result is that alliance's score. To determine the winner, it sees who is larger.

Dezion 28-04-2016 22:32

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1579927)
What we can tell you is that our match prediction system (in a state less accurate than it is now,) was able to predict 6 out of 7 of the advancements in the finals, including predicting correctly lower seeded alliances triumphing over higher seeds, a victory even we humans didn't quite predict.

I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?

weaversam8 28-04-2016 22:50

Quote:

Originally Posted by Dezion (Post 1580188)
I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?



Sorry, I may have not given enough information there. Those predictions were based on the data we had up to that point, and not after, with the data we have now. If the bots in eliminations then had the capabilities they do now, it might have turned out different. We were also using a different ranking system at the time, our new system has been adjusted, but may not apply to some of our older results.

TomAwezome 28-04-2016 23:07

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Dezion (Post 1580188)
I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?

Eliminations has different scoring, and as of right now it is applying the points as if it is a qualification match. We will look into getting that functional as soon as possible. Thanks!


Edit: My programming team lead (Sam; weaversam8) has informed to let you know that he is tired and delirious and that you should only pay attention to this response to your question, not his.

TomAwezome 29-04-2016 10:54

ORB back-end API public release!
 
As ORB's back-end nears completion, we are releasing access to it! This is what the currently accessible front-end uses to get the info. There are just a small few of endpoints that are not implemented, and as such will not be listed, but in the future they will be added. Otherwise, everything else is currently functioning as intended! (A side note, as mentioned above, scoring for anything that is not a qualifying match is currently wrong as time has not been available to carve the algorithm for the different scoring playoffs uses, but before World's playoffs start we will have it ready.)

http://orb.scoutfrc.io/


ORB API Endpoints:

GET /list/<eventcode> — Returns a list of teams at event identified by eventcode.

GET /team/<number> — Returns a basic JSON object about that team.

GET /team/<number>/defense — Returns the entire defense skill lineup for that team.

GET /team/<number>/defense/:defensenumber — Returns the defense skill for that team on defense X.

GET /team/<number>/goals/high — Gets a teams skill at a high goal in teleop/auto.

GET /team/<number>/goals/low — Gets a team's skill at a low goal in teleop/auto.

GET /team/<number>/scale/ — Gets a team's skill as a percent decimal (0 to 1) at scaling.

GET /team/<number>/challenge/ — Gets a team's skill as a percent decimal (0 to 1) at challenging.

GET /team/<number>/score — Returns the team’s score for rankings.

GET /work/match/<eventcode>/<matchidentifier>/ — Calculates result of a match by comparing teams’ scores, and ranking alliance scores, returns two values, first is red's score, second is blue's score.

GET /work/defense/<eventcode>/<matchidentifier>/ — Calculates the optimal defense selection for both alliances, returns array with arrays for red and blue, arrays in those: first array is defenses in defense categories they are best at, second is array for defenses in defense categories they are worst at.

Edit: event code and match identifier formats are the same as TBA's.

weaversam8 30-04-2016 09:41

Re: Project ORB: A superb predictive scouting system!
 
Good morning folks, and congratulations to all the teams who were selected for eliminations matches (we were not.)

As an update, and a testament to ORB's match prediction abilities, we are going to try and run some statistics to predict advancement of alliances in each division here at champs. I'll keep you up to date as much as we can.

weaversam8 30-04-2016 10:36

Re: Project ORB: A superb predictive scouting system!
 
Proud to announce that quarterfinals predictions are available in ORB as of now. It's important to realize that each match in a series (i.e: Q1M1,Q1M2,Q1M3) are all predicted with the same score based on how our system works. This is the best we can do. Therefore, if you look at the winning score for each Quarterfinal Series, you can see the predicted advancing alliance by looking at the winner.

Here are quick links in case you don't want to navigate the system (scroll to the bottom):
Archimedes
Carson
Carver
Curie
Galileo
Hopper
Newton
Tesla

weaversam8 01-05-2016 09:57

Re: Project ORB: A superb predictive scouting system!
 
After this weekend, I'd like to thank the 118+ teams that used Project ORB. This was a very useful tool for us, and an excellent learning experience. We hope you found it useful!

We plan on continuing Project Orb in the fall, so please feel free to follow up with us as next year's season approaches. I'm sure we all need some time to recover :) .

I'm glad we've all had a great time sieging the Stronghold- we'll see you next year!!

<3 Wired Wizards

Foster 02-05-2016 06:40

Re: Project ORB: A superb predictive scouting system!
 
Can you post how well your predictions worked against actual results?

weaversam8 04-05-2016 18:53

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Foster (Post 1581477)
Can you post how well your predictions worked against actual results?

We haven't done the math for it yet, we're all still pretty tired and busy with exams. The API endpoints our programmer Tom posted above expose all the data, and it hasn't been retrained since before Worlds, so you are welcome to run the numbers yourself if we can't do it any time soon.

Brian Maher 13-07-2016 20:17

Re: Project ORB: A superb predictive scouting system!
 
Would it be possible to prepare ORB numbers for IRI? This system is really cool, and with IRI looming, I was wondering if we could see ORB output for the IRI teams?

Dibit1010 13-07-2016 20:59

Hi Wired Wizards, I'm from team 587 the hedgehogs- are you planning on publishing data during events in real time or are you guys planning to keep if for yourselves?


Sent from my iPhone using Tapatalk

weaversam8 28-07-2016 19:04

Re: Project ORB: A superb predictive scouting system!
 
Hey guys!

Unfortunately, our backend was taken down after the season ended to save costs. Otherwise we'd run the numbers, of course. Now, what I can do for you, if your team is really serious about doing analysis, is link you to the server source code. It's very messy and unclean, but we don't have programming members able to work on it currently.

Going forward, we plan to maintain ORB, and yes, we plan on continuing to publish data live during events.

I'll make a post in a few minutes with the source.

weaversam8 28-07-2016 19:49

Re: Project ORB: A superb predictive scouting system!
 
Good evening everyone!

As requested, we are publishing the source to the ORB backend. It is my regret to publish the source, as it is lacking documentation, and not clean, but I have faith in the FRC community, that if someone really wants to use it, they'll be able to, and we'll attempt to provide the best support for this legacy code that we can.

Here's a snippet from what I posted in the repository readme:
Quote:

It is my pleasure to share the culmination of our efforts at Neural Network scouting using our ORB system. It was not my intention to share this code in such an unorganized form, but we do not have the resources to thoroughly document this code at the present time. Demand has risen for this code, so we've decided to release it in its current form. Please open PRs and issues on this repository directly, and we'll try and get to them as best we can. This system was designed for FIRST Stronghold, and predictions are to be taken with a grain of salt. Team 4534, the Wired Wizards, hopes to continue to support ORB through future FRC games, and we plan to provide the same standard of service we provided during the world championship all season long. In the mean time, feel free to use this code for offseason events at your own discretion. Pair this with the orb-client repo, and you'll really have something. We hope you enjoy ORB!
Here's the repo URL, be sure to submit PRs and issues as you find them, and we'll attempt to get back to them as soon as we can.
https://github.com/4534-WiredWizards/orb-backend

Have fun folks!

marshall 29-07-2016 10:58

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1598970)
Good evening everyone!

As requested, we are publishing the source to the ORB backend. It is my regret to publish the source, as it is lacking documentation, and not clean, but I have faith in the FRC community, that if someone really wants to use it, they'll be able to, and we'll attempt to provide the best support for this legacy code that we can.

Here's a snippet from what I posted in the repository readme:


Here's the repo URL, be sure to submit PRs and issues as you find them, and we'll attempt to get back to them as soon as we can.
https://github.com/4534-WiredWizards/orb-backend

Have fun folks!

Thanks for sharing!!!


All times are GMT -5. The time now is 12:53.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi