Neural Network Approach

Hello, 604’s programmer here. A lot of the scouting “algorithms” and “predictions” I’ve seen have been based on using algorithms or selective data. I’ve been thinking of using a neural network approach. I have a multi-layer perceptron neural network using back-propagation ready to go, and all I need now is data. I don’t want to use data from TBA, as that data is based on alliance performance, rather than that of individual teams. I need data which will give me a list of teams, along with their auton gears, average teleop gears, average kPa, and climb percentage, if possible. The goal of the neural network is to predict not only what teams would win when matched up against other teams, but also to calculate how exactly synergy works when it comes to alliance cooperation.

This has probably been attempted before, but I would like to try this myself, and we would collectively be able to generate a better algorithm if we were to share data with each other. If I get anything substantial, I will let you all know (and post the neural network and its weights, along with a program pre-loaded with the weights for prediction) on github.

Thank you <3

Team 8 has data on most of the things you described for teams at SVR. We would be happy to share it with you through PM.

That sounds fantastic for a scout to have such a tool…however the data collection you seek is nearly all observational for separate individual performance data. Also its per event basically.

The way Steamworks tracks teams:

Auto mobility score is tracked per team but not posted (added rotor auto score per alliance) total points in auto posted per alliance.
Rotors in tele-op are per alliance
Fuel in tele-op is per alliance
Yellows/Reds are per team (except in eliminations then per alliance)
Ready for takeoff is per alliance.

That’s why the raw stats are simply competition alliance based totals of the categories and the “individual” scores/metrics posted are always based majorly on partner play as well.

Now in terms of a scouting neural-net app. First off you would need to do individual specific observations to count gears, fuel , cycle times etc for each and every team individually to have the data you are seeking. Is that feasible? Yes we do that with pen, paper and highlighters.

Otherwise you pretty have to accept that any posted data will not be based on the individual team rather the alliances they were a member of in that competition based on scheduling . I don’t see any way to get the individual data you are seeking unless you watch and track every match and every team . If you do that you already know (or should know) who’s good and who’s not, not sure if an algorithm could do better than a group human scouts taking notes . That in essence is what we do…its called qualitative scouting and yes we watch every match and take notes on every bot including gears, climbs , fuel, hp. fouls, liabilities, tendencies etc.

Since its a team based competition you can safely assume the numbers available are somewhat skewed and somewhat luck based. This is a key reason why we choose not to use posted numbers as a primary ranking mechanism, rather we simply use them to back up our observations and make sure we did not miss anyone our eyes/brain told us were good. In two competitions we added one bot based on day 1 numbers in day 2 out of 54 bots (27-deep each event) on our observation based pick lists, after we watched them closer day 2 they did not impress enough to break onto our list…whether captain or not these lists help so we make them up every competition.

I saw Team 8’s scouting app in Ventura as they were our pick ,its amazing perhaps they do have the individual data you seek. I was impressed.

I could give you some scouting data from the two regionals my team attended, but that’s about all I can offer you as far as single team performance. You might just find more success analyzing OPR, DPR, and CCWM like TBA does and then maybe adding some more heuristics and calculating them yourself.

I have toyed with this concept for a while now, I am glad to see people have taken a stab at this recently.

I have a few issues with this approach to find your perfect match at an event, or for match predictions for that matter.

  1. Lack of data. It takes a lot of data to train up an ANN, and frankly, you don’t have that available to you on scouting meeting night. As has previously been mentioned a lot of the data you will have to be adding yourself, not pulling from the API. This issue is not insurmountable but seems plenty annoying, your time may be better spent working on your scouting meeting agenda.

  2. **Using this going into an event. ** While it may be possible to run this sort of thing going into #2champs you still need to find a way to normalise all the data for training, not easy when the number of matches a team has played to get to worlds varies from 8 - 60+. Again not insurmountable, but highly likely to cause significant skews.

  3. Using old data. As tempting as it is using data from a previous season as a basis is probably a very bad idea. 'nuff said.

Here’s what I suggest pursuing as someone that has very minimal experience with ANNs and Expert Systems. Try training to eliminate the bottom tier teams from your pick lists. I know, this may seem trivial, but it can be a huge help, especially when looking for a 2nd or 3rd pick. With this approach, (and varying your set sizes dependent on your current needs) you can pull the cream of the crop so to speak from your data, and rank those with good old scouting meeting discourse. This is probably where you will see your biggest time savings with any sort of filtering.

I am not sure how well this approach (ANN in general) would fair in a real-time application, such as during alliance selections. I wouldn’t put my chips on it.

If anyone attempts this approach I would be fascinated by the results, so please do share.

Skye Leake

Was going to say this, so I’ll leave another algorithm for you to look into: support vector machines.

I understand this, and it’s why I am asking for individual data, rather than pulling the data directly from TBA (which I am still doing as a practice exercise). However, my team also took extremely detailed individual scouting data for the same two events (so proud of our sophomore scouting lead he’s a downright workaholic), so I won’t be needing data for sfr or svr.

This type of data would be helpful.

While it is a good idea, I think in a game such as this one (in which the outcome is subject to hidden factors in the statistics, such as team synergy), it would be more beneficial to have the option of including hidden layers and multiple outputs using a parametric ANN algorithm versus an SVM one.

Ah, I had the same concerns, but I have already come up with a solution: instead of using a team’s statistics as a whole, simply use their individual statistics as separate input nodes when feeding into the network. This way, you can effectively turn every single FRC event this season into a single database to use as your input, which effectively provides me with quite a bit of freedom in what data I can use. And you are indeed correct regarding the normalization of data, but the idea is that you wait until elims, and plug in the values you found for the different teams, and then use that to generate an optimal alliance, or predict eliminations results. In addition, you could use a team’s data from previous regionals this season and use that as a rough prediction.

Just an update; I’ve started working with Eugene Fang (you know, the guy who built TBA) and so now I can use data dumps to train my program (with TBA numbers, for the time being, and see if I can get anywhere). I’ll start working on it perhaps this Thursday, and I’ll show whatever results I get here.

We’ve got some of the data you mentioned. I’ll send it over tomorrow.

When you actually have something that makes predictions, let me know. I would love to compare your predictions to my Elo and predicted contribution match predictions. I have seen similar efforts in the past and many of them severely suffer from overfitting problems, so make sure to keep your training and testing data mutually exclusive.

Again I have data normalisation concerns… as for using previous events; some teams have 2 matches in elims, others have 9+ matches…

As you brought up earlier, there is going to be a lot of work done in order to adjust the weights, and some back propagation of weights to expedite training. This may be a good thing to test ex post facto on sub-division finals and Einstein due to a large number of matches being played with the same alliances. (although it may suffer from some of the score maximisation points we have seen thus far in the season, i.e. 445 (sorry for bringing that number up CalTran :P)).

I think where the ANN really would excel is not in trying to assemble an optimal alliance, but trying to predict who seeds will pick which based on individual team strengths/shortcomings and the diversity of the teams at the event. Piecing together alliances to reverse engineer strategy is not always the easiest task, especially this year. This would be HUGE on scouting night to identify threats to your alliance from other alliances.

Skye Leake

I’ve played with using an ANN that maps a vector containing the sums of an alliance’s component OPRS to a single output score, and using the default settings in sklearn gives me a root mean squared error of about 42 points for qual matches, plus a Brier score of around .185 for match predictions. I’m sure somebody with more knowledge of ML (and more computational power) than I do could create one that performs better.

For 2017 data?

Yes. The one I made uses the alliance’s combined component OPRs for autoFuelPoints autoRotorPoints, teleopFuelPoints, teleopRotorPoints, and teleopTakeoffPoints to generate scores. I’ve not tried it on any other year.

Brier score of 0.185 and RMS score error of 42 points is INSANELY good, especially for 2017. Would you mind sharing your method/code in more detail? I’m actually slightly dubious of the results since in my experience, penalties introduce a ton of variance that’s really difficult to predict.

I’ll second this. I’ve probably spent more time building predictive models for this year than anyone, and my very best models in ideal scenarios get an RMS score error of 56 and a Brier score of 0.203.

How big was your sample size and were your training/testing data mutually exclusive?

Did you separate your training and testing data? If not, this accuracy could simply be a result of overfitting. Have you tried using the generated weights to predict match outcomes for other regionals which weren’t used as part of the training data set?

If so, I would very much like to know how many hidden layers you used, what activation function you chose, as well as how you structured your network in general.