Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Championship Event (http://www.chiefdelphi.com/forums/forumdisplay.php?f=12)
-   -   Project ORB: A superb predictive scouting system! (http://www.chiefdelphi.com/forums/showthread.php?t=147817)

asid61 28-04-2016 00:32

Re: Project ORB: A superb predictive scouting system!
 
The website is showing the "loading teams..." message for an awfully long time for me (it never loads the teams :(). Should I be worried?

EDIT: Carver loads fine, for some reason, although there is still a delay in opening.
DOUBLEEDIT: Also, what do the numbers mean? Is it all relative measurement?

weaversam8 28-04-2016 00:34

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by weaversam8 (Post 1579933)
We noticed the error in the logs before you reported it- we're looking into it right now. It seems that we're missing data points for 3 out of 3144 teams, and one of them happens to be in Curie...

We have discovered that we're missing data for Teams 4, 8, and 11 for scaling. We're going to inject artificial data for those teams (just for that single value) tonight, and tomorrow update it with the most accurate data. That's all we can do tonight.

Kyler Hagler 28-04-2016 00:43

Re: Project ORB: A superb predictive scouting system!
 
Looks like it is working now. Great job getting it up. Thanks for the work and dedication this took and allowing it to be usable for everyone.

microbuns 28-04-2016 01:23

Re: Project ORB: A superb predictive scouting system!
 
This is incredible. I love the fact that I can still pull up data for teams that are not at champs. The data looks pretty darn accurate. I have two questions:

1) Will these be updating as new matches are played during champs?
2) Could you release a top 50/100/3000? I'd be interested to see how your program ranks the robots, and compare it to OPR and my personal favorites. I could (and might) compile this myself, but it'd take a while, and you might have some way of doing it much more quickly.

GKrotkov 28-04-2016 06:22

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by BMSOTM (Post 1579760)
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.

I second this like nobody's business. I would love to compare ORB with 1712's scouting data for MARCMP.

weaversam8 28-04-2016 06:59

Re: Project ORB: A superb predictive scouting system!
 
Good morning everybody!

Quote:

Originally Posted by microbuns (Post 1579949)
This is incredible. I love the fact that I can still pull up data for teams that are not at champs. The data looks pretty darn accurate. I have two questions:

1) Will these be updating as new matches are played during champs?
2) Could you release a top 50/100/3000? I'd be interested to see how your program ranks the robots, and compare it to OPR and my personal favorites. I could (and might) compile this myself, but it'd take a while, and you might have some way of doing it much more quickly.

We do plan on updating based on matches being played, not sure if it's live this morning, but if it's not, it will be as soon as possible. Also, I'll take note of your request about the top lists, we'll see if we can do it. It might interest you to understand our match scoring system:

Match Scores
For match scores, our system takes the highest skill level of each individual defense and assumes that is the score of the robot crossing the defense, then it uses that for purposes of scoring. Then, it calculates in low and high goal shots, and then calculates the scale/challenge bonus.

Quote:

Originally Posted by BMSOTM (Post 1579760)
This is really interesting; I can't wait to see how well it predicts at CMP. In the meantime, is it possible you could release its output for some of the other competitions this season? I'd really appreciate if you could release its evaluation of the teams at MAR CMP in particular so I can see how accurate the results are.

In regard to releasing previous data:
I can do a data dump of whatever teams you like, including the ones at the MAR Championship, but right now, data in the system will be skewed by the fact that the matches at that championship became part of the team's score, which means it could be less truly accurate. We might have the option to release some data from earlier in the season, but we'd have to dedicate training resources for that, which may be one of our lower priorities at the moment. We'll see what we can do.

Coming next:
  • We're looking into building a separate match ranking system that only takes this event into account, like how OPR individually examines an event.
  • Top X teams list? (You can already kindof do this in a division by sorting)
  • Slackbot to interface with the ORB API
  • Recommended defenses for a match view
  • And more...

kinganu123 28-04-2016 09:53

Re: Project ORB: A superb predictive scouting system!
 
How is the data represented for the neural network? I was thinking about this the other day, and wasn't too sure how you'd account for individual teams without essentially combining the teams' stats into one "super team" (for a lack of a better phrase).

Eugene Fang 28-04-2016 10:30

Re: Project ORB: A superb predictive scouting system!
 
I haven't looked at how accurate ORBs predictions have been, but I noticed that it is consistently overestimating scores by a large margin. Any insight as to why this might be occurring?

CrazyMohawk 28-04-2016 10:38

Re: Project ORB: A superb predictive scouting system!
 
its down

danielrw 28-04-2016 10:41

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by CrazyMohawk (Post 1580044)
its down

Thanks, we're working on getting it back up.

TomAwezome 28-04-2016 10:44

Re: Project ORB: A superb predictive scouting system!
 
Howdy!! I'm Tom, Wired Wizards' Alpha Team leader (Robot Code), and I as well have been doing the vast majority of the back end for ORB.

Quote:

Originally Posted by kinganu123 (Post 1580024)
How is the data represented for the neural network? I was thinking about this the other day, and wasn't too sure how you'd account for individual teams without essentially combining the teams' stats into one "super team" (for a lack of a better phrase).

The data from the neural networks is saved into a database into tables for goals, defenses, challenge, and scale; goals has 4 values (autolow, autohigh, teleoplow, teleophigh), defenses has 9 values (0-8, 0:lowbar, 1:portcullis, 2:cdf, 3:moat, 4:ramparts, 5:drawbridge, 6:sallyport, 7:rockwall, 8:roughterrain), scale and challenge each have 1 value for their respective percentage. The goals values are representative of the quantity of goals, defenses representative of 0-2 crossings (as that is all TBA has per match), scale and challenge as mentioned are stored as a value between 0-1, a decimal percent. As each match is a combination of 3 teams per alliance, pulling the data gives you an idea of how they perform, but of course it is just a showing of how the alliance performs! The magic happens as the networks for each team analyze all of their different combinations, and finds the trends inside the dataset that would indicate the team it's training for. Good question!

danielrw 28-04-2016 11:05

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by danielrw (Post 1580045)
Thanks, we're working on getting it back up.

We restarted the server and everything seems to be working now.

marshall 28-04-2016 11:57

Re: Project ORB: A superb predictive scouting system!
 
Source code?

TomAwezome 28-04-2016 12:10

Re: Project ORB: A superb predictive scouting system!
 
Quote:

Originally Posted by Eugene Fang (Post 1580042)
I haven't looked at how accurate ORBs predictions have been, but I noticed that it is consistently overestimating scores by a large margin. Any insight as to why this might be occurring?

As far as I could tell much earlier in development, it was oddly over in estimation, but it seemed consistent by a certain multiplier (not impacting the predictions for winning, just showing bigger values.) Before, I had multiplied it by about .7310 to get the scores looking more like actual match scores. It is currently not doing that, since as mentioned it has no impact on prediction. The score value is a result of how it's summing it all, as it is trying to follow as close as possible the way that match scoring works in-game, but is not caused by any errors.

weaversam8 28-04-2016 13:50

Re: Project ORB: A superb predictive scouting system!
 
Hope everybody's been having a great day! We have found that there are some issues with retraining teams as matches occur, so we've turned off that feature in order to ensure that the web service remains online. We'll train tonight for all teams at Championships to make sure the data tomorrow is fresh based on how teams played today.

Quote:

Originally Posted by TomAwezome (Post 1580068)
As far as I could tell much earlier in development, it was oddly over in estimation, but it seemed consistent by a certain multiplier (not impacting the predictions for winning, just showing bigger values.) Before, I had multiplied it by about .7310 to get the scores looking more like actual match scores. It is currently not doing that, since as mentioned it has no impact on prediction. The score value is a result of how it's summing it all, as it is trying to follow as close as possible the way that match scoring works in-game, but is not caused by any errors.

We just did a tweak and they should be slightly more realistic, but we'll be looking into adjusting these values tonight to make them appear more like match scores.

Quote:

Originally Posted by marshall (Post 1580062)
Source code?

We plan on releasing the source code at some point, it's a bit fragmented at the moment and we'll need to clean it up.


All times are GMT -5. The time now is 12:53.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi