Project ORB: A superb predictive scouting system!

[microbuns]

Originally Posted by GKrotkov

I second this like nobody's business. I would love to compare ORB with 1712's scouting data for MARCMP.

MARCMP. In order to get a specific event, go to the Blue Alliance, find the event you want, and navigate to its event page. In the URL, after the last slash, there will be the "event code" - something like 2016mrcmp (for MAR champs). Take this event code, and place it in this URL:

http://orb.scoutfrc.com/#/a/event/<EVENT CODE>

[kinganu123]

Originally Posted by TomAwezome

Howdy!! I'm Tom, Wired Wizards' Alpha Team leader (Robot Code), and I as well have been doing the vast majority of the back end for ORB.



The data from the neural networks is saved into a database into tables for goals, defenses, challenge, and scale; goals has 4 values (autolow, autohigh, teleoplow, teleophigh), defenses has 9 values (0-8, 0:lowbar, 1ortcullis, 2:cdf, 3:moat, 4:ramparts, 5:drawbridge, 6:sallyport, 7:rockwall, 8:roughterrain), scale and challenge each have 1 value for their respective percentage. The goals values are representative of the quantity of goals, defenses representative of 0-2 crossings (as that is all TBA has per match), scale and challenge as mentioned are stored as a value between 0-1, a decimal percent. As each match is a combination of 3 teams per alliance, pulling the data gives you an idea of how they perform, but of course it is just a showing of how the alliance performs! The magic happens as the networks for each team analyze all of their different combinations, and finds the trends inside the dataset that would indicate the team it's training for. Good question!

Ok, so my followup question is how are you combining the teams into an alliance? Do you use something like CCWM to weight each team into a single alliance?

[TomAwezome]

Originally Posted by kinganu123

Ok, so my followup question is how are you combining the teams into an alliance? Do you use something like CCWM to weight each team into a single alliance?

The only data it uses is data it has pulled then trained. To compute how a combination of teams adds up as an alliance, it, for each defense, finds the max of the three teams' defense scores for that defense, then multiplies that by either 5 for lowbar and 2.5 for the other defenses. The same for goals, but with the proper point values for auto/teleop high/goal. And again, the same for scale and challenge, with the proper score multipliers. It then adds these all together and the result is that alliance's score. To determine the winner, it sees who is larger.

[Dezion]

Originally Posted by weaversam8

What we can tell you is that our match prediction system (in a state less accurate than it is now,) was able to predict 6 out of 7 of the advancements in the finals, including predicting correctly lower seeded alliances triumphing over higher seeds, a victory even we humans didn't quite predict.

I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?

[weaversam8]

Originally Posted by Dezion

I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?

Sorry, I may have not given enough information there. Those predictions were based on the data we had up to that point, and not after, with the data we have now. If the bots in eliminations then had the capabilities they do now, it might have turned out different. We were also using a different ranking system at the time, our new system has been adjusted, but may not apply to some of our older results.

[TomAwezome]

Originally Posted by Dezion

I'm confused here. According to what I can see at this page, the only Eliminations matches ORB was able to correctly predict was 2v7 and the Finals. Am I interpreting something incorrectly?

Eliminations has different scoring, and as of right now it is applying the points as if it is a qualification match. We will look into getting that functional as soon as possible. Thanks!


Edit: My programming team lead (Sam; weaversam8) has informed to let you know that he is tired and delirious and that you should only pay attention to this response to your question, not his.

[TomAwezome]

As ORB's back-end nears completion, we are releasing access to it! This is what the currently accessible front-end uses to get the info. There are just a small few of endpoints that are not implemented, and as such will not be listed, but in the future they will be added. Otherwise, everything else is currently functioning as intended! (A side note, as mentioned above, scoring for anything that is not a qualifying match is currently wrong as time has not been available to carve the algorithm for the different scoring playoffs uses, but before World's playoffs start we will have it ready.)

http://orb.scoutfrc.io/


ORB API Endpoints:

GET /list/<eventcode> — Returns a list of teams at event identified by eventcode.

GET /team/<number> — Returns a basic JSON object about that team.

GET /team/<number>/defense — Returns the entire defense skill lineup for that team.

GET /team/<number>/defense/:defensenumber — Returns the defense skill for that team on defense X.

GET /team/<number>/goals/high — Gets a teams skill at a high goal in teleop/auto.

GET /team/<number>/goals/low — Gets a team's skill at a low goal in teleop/auto.

GET /team/<number>/scale/ — Gets a team's skill as a percent decimal (0 to 1) at scaling.

GET /team/<number>/challenge/ — Gets a team's skill as a percent decimal (0 to 1) at challenging.

GET /team/<number>/score — Returns the team’s score for rankings.

GET /work/match/<eventcode>/<matchidentifier>/ — Calculates result of a match by comparing teams’ scores, and ranking alliance scores, returns two values, first is red's score, second is blue's score.

GET /work/defense/<eventcode>/<matchidentifier>/ — Calculates the optimal defense selection for both alliances, returns array with arrays for red and blue, arrays in those: first array is defenses in defense categories they are best at, second is array for defenses in defense categories they are worst at.

Edit: event code and match identifier formats are the same as TBA's.

[weaversam8]

Good morning folks, and congratulations to all the teams who were selected for eliminations matches (we were not.)

As an update, and a testament to ORB's match prediction abilities, we are going to try and run some statistics to predict advancement of alliances in each division here at champs. I'll keep you up to date as much as we can.

[weaversam8]

Proud to announce that quarterfinals predictions are available in ORB as of now. It's important to realize that each match in a series (i.e: Q1M1,Q1M2,Q1M3) are all predicted with the same score based on how our system works. This is the best we can do. Therefore, if you look at the winning score for each Quarterfinal Series, you can see the predicted advancing alliance by looking at the winner.

Here are quick links in case you don't want to navigate the system (scroll to the bottom):
Archimedes
Carson
Carver
Curie
Galileo
Hopper
Newton
Tesla

[weaversam8]

After this weekend, I'd like to thank the 118+ teams that used Project ORB. This was a very useful tool for us, and an excellent learning experience. We hope you found it useful!

We plan on continuing Project Orb in the fall, so please feel free to follow up with us as next year's season approaches. I'm sure we all need some time to recover .

I'm glad we've all had a great time sieging the Stronghold- we'll see you next year!!

<3 Wired Wizards

[Foster]

Can you post how well your predictions worked against actual results?

[weaversam8]

Originally Posted by Foster

Can you post how well your predictions worked against actual results?

We haven't done the math for it yet, we're all still pretty tired and busy with exams. The API endpoints our programmer Tom posted above expose all the data, and it hasn't been retrained since before Worlds, so you are welcome to run the numbers yourself if we can't do it any time soon.

[Brian Maher]

Would it be possible to prepare ORB numbers for IRI? This system is really cool, and with IRI looming, I was wondering if we could see ORB output for the IRI teams?

[Dibit1010]

Hi Wired Wizards, I'm from team 587 the hedgehogs- are you planning on publishing data during events in real time or are you guys planning to keep if for yourselves?


Sent from my iPhone using Tapatalk

[weaversam8]

Hey guys!

Unfortunately, our backend was taken down after the season ended to save costs. Otherwise we'd run the numbers, of course. Now, what I can do for you, if your team is really serious about doing analysis, is link you to the server source code. It's very messy and unclean, but we don't have programming members able to work on it currently.

Going forward, we plan to maintain ORB, and yes, we plan on continuing to publish data live during events.

I'll make a post in a few minutes with the source.