Why do you use match predictions

#1

Purpose is exactly as the title states (plus a little more), for those of you that make match predictions from your scouting data, why do you do this and why do you find it useful? 78 has been making our scouting and strategy more robust and we are looking to see if match predictions could be valuable. When we looked through 1678’s 2018 scouting white paper they did a ton of predictions, everything from ranking points, seeding, number of climb/park points etc. If anyone can share from experience why your team would use predictions, we would love to hear about it.

If we want to learn more about the process of computing the predictions, do you have somewhere for us to start?

#2

One thing we use them for is when we are strategizing for matches. If we are predicted to lose by a decent margin, it could be advantageous to chose a higher risk strategy.

This also applies to ranking where if you are in a race for a high spot, if the team you are competing against is predicted to get lots of bonus RPs, you may need to go for the as well whereas if your opponent isnt predicted to get many bonus RPs, you can maybe play matches more safely and just “take the win”.

10 Likes
#3

604 uses match predictions primarily for match strategy and for planning out alliance selections.

@Conallking already mentioned the idea of a high-risk/high-reward strategy for predicted losses and going for the “guaranteed RP” to secure a certain ranking.

For alliance selections, we use predictions to narrow down the different ranking possibilities so we aren’t surprised if the top 8 rankings change during the last day of qual matches. That way, we can spend more time evaluating likely scenarios when it comes to picking (e.g. avoiding the “strong” side of the bracket, dealing with scorching, etc.).

The Blue Alliance has predictions at a hidden URL for each event (e.g. https://www.thebluealliance.com/event/2019casj/insights). We compute a slight variant of OPR for each team (but you can just think of them as OPRs), then add the OPRs for teams in an alliance to predict their score. Depending on the relative predicted scores of the Red and Blue alliances in a match, we can be more or less certain that one alliance will win over the other. This OPR method works better in some years than others depending on the game, but usually yields around a 70% prediction accuracy. We also compute component OPR for stats relevant to bonus RP, and throw all of that into a Monte Carlo simulation to predict rankings. This was all stuff I developed back in 2016, but I haven’t had the chance to make it better, so there’s definite room for improvement.

@Caleb_Sykes’s Event Simulator (Sykes Event Simulator 2019) uses a method that is pretty similar to TBA’s, but also includes a team’s Elo as part of the prediction instead of just OPR (and is probably tuned way better than TBA’s). I’ll let him chime in on the details if he wants, before I say anything incorrect.

There’s also usually some interesting discussion in the “Match Prediction Contest” threads (e.g. 2019 Houston Match Prediction Contest) about different prediction methods the community is trying out.

Hope this!

9 Likes
#4

EDIT - following title, not necessarily OP :blush:

As a spectator, I like to use predictions for these main reasons:
Determine reliability of popular models to a particular event
Identify exciting matches to watch (where there’s not necessarily a rooting interest)
Follow my team’s performance versus prediction through the event to know how well they actually did.

#5

4276 has used a score prediction, not sure if you’re call that the same as “match prediction”. But we look at average scoring contributions of teams and give them a best case alliance score. If that is lower than our alliance’s predicted score, we can play less defense. If it is higher, we need a dedicated defender to block their high scorer.

If your scouts are keeping good records on # of hatches and cargo scored per match you can calculate averages for each team.

1 Like
#6

I’ve used them in the past to simulate strategic choices for robot designs. In general I’ve found that to be too much effort though.

#7

Thank you for the great information. Pertaining to TBA predictions, I am slightly confused at this graph

11%20PM
Screen Shot 2019-10-08 at 1.01.11 PM.png1108×198 25.2 KB

Could you tell me what exactly I am looking at. I believe x axis is score, with the vertical bars being predicted score? What are the curved lines though?

#8

I think the bell curve is the predicted score range, with the vertical line being the actual score. In the match you pictured, TBA expected the blue alliance to get 39 points, but they actually got 32, where they expected red to 56 pts but they actually got 74.

#9

Yeah I switched that up, that makes more sense.

#10

This is correct. The score distribution doesn’t actually follow a bell curve, but the math makes some basic approximations and it’s shown like that for visualization purposes.

2 Likes
#11

I like to look at match predictions for a couple reasons

  1. Provides a second opinion beyond our intuition and our data on our risks in the match, which can inform whether we play a risky or safe match strategy.
  2. Rankings simulations (which are usually aggregated from match simulations) can be incredibly helpful for focusing on preparing for the most likely scenarios
#12

I could see how match prediction could be helpful later in the event for potential seeding implications. This past year we had a very simple sum of each robots points to get a rough idea of how easy or hard the match would be. It was nice to share with the non-scouts that ask “how does our next match look”, but we generally dig deeper into the data for match strategy (ie: team may have climbed 2/5 matches but climbed in last two matches, so they are more likely than 40% chance to climb). Having an idea of how hard or easy a match is helpful to determine if how risky of a strategy to pursue and/or if you can spend time on bonus RPs, but I don’t think you need match predictions to determine this.

#13

How often are those predictions updated?

1 Like
#14

I think others have covered the main reasons, so I’d like to explore some additional reasons why I believe match predictions can be helpful.

Plenty have already mentioned risky/conservative win strategies, and that’s important for sure, but remember that there are two other RPs in the match as well from bonus objectives. As a rule of thumb, I would always highly prioritize the bonus RPs in matches where the outcome was >70% certain. That’s because when you get into matches with that large of a skill gap between the alliances, there’s not a lot to be gained by either alliance from focusing on winning as the outcome has essentially already been scheduled. If that happens, just take the win/loss and go guarantee you get as many bonus RPs as you can.

The other thing I’d like to point out regarding match predictions is that they are the best validation tool you have for your scouting system. When you get around to alliance selection, you are trying to pick the teams that give you the best chance of winning. I really don’t care how high they score on all of your fancy scouting metrics, what matters is how well your scouting can actually predict who will win and lose.

Honestly, if you’re not validating your scouting with match predictions or something similar, you are becoming dissociated with reality. It’s easy to look back on matches that have already happened and explain why the outcome ended up the way it did, it’s much harder, but much more enlightening imo, to try to look forward and predict what will happen. This can be a bit more painful of a process, as you’ll be forced to face your own incorrect predictions head on, but that’s really what you gotta do if you really want to learn and improve. For reference, I can predict the winner of FRC matches about 70% (year dependent) of the time using Elo/OPR, so you should be hitting at least that high using your scouting data or you’re probably doing something wrong.

Sure! Here’s a good jumping off point. You can use a simple logistic function to make win probabilities. For each team, give them a rating (I’ll be assuming the rating has units of 2019 points, but any units can work), which we’ll denote q. Sum the ratings for the red robots to get q_red, and likewise for the blue robots to get q_blue. Plug those ratings into the following formula:
WP_blue = 1/(1+10^((q_red - q_blue)/s))
Where WP_blue is your predicted blue win probability and s is a scale factor that you have to determine. 30 is a good baseline for s in 2019, go higher if you have weaker scouting data or are early in the event, go lower if you are more confident in your scouting data or have verified predictions at earlier events.

Play around with prior scouting data and match results to determine how exactly to make your team ratings, a good starting point is just to sum up a team’s average points scored across all categories based on your scouting data, but you can add or subtract from teams’ ratings using whatever criteria you want. Understand this is just a baseline, you can start to play around with alternative formulas or variants of this one once you feel like you understand this formula well enough and it feels too restrictive.

Finally, I know Eugene already mentioned it, but jump on in to some of the prediction contests. It’s a fun way to compare your results to others and you can ask questions and learn right along with everyone else. I’m happy to answer any questions about my event simulator or Elo if you have any.

6 Likes
#15

Maybe we have different definitions of “scouting system”, but wouldn’t validating against FMS scoring be more accurate?

A lot of what say makes sense in theory, but I think we are a long way from getting to that point where match prediction plays a role in alliance selection. It would be cool if each robot had a ‘winning percentage if picked’, similar to how analytics is used in the NFL for 4th down decisions (ie: 17% chance if punt, 19% if go for it), but I’m not aware of team currently using that. I also question if that is even possible given the small sample sizes – due to both matches at event and yearly changing game.

#16

Not quite sure what you mean here, let me try to rephrase my statement a little better:

The best way to validate your scouting data is to make predictions for upcoming matches and compare those predictions to the actual results of those matches. Those predictions can be any of simple W/L predictions, win probabilities, final score predictions, or full score breakdown predictions.

The two key points for me are that:

  1. you are comparing something forward looking (predictions)
  2. with a measurable objective result

And the difference between 1 and 2 should be minimized.

#17

I think we are talking about two different things.

I’m referring to validating scouting data that says that 254 climbed in match 12. In order to validate that 254 actually climbed that match, I think checking the FMS data would perfect.

I think you are getting at validating the value of the scouting data. The idea being that even if we had perfectly accurate match data, if it doesn’t predict future matches, then the scouting data isn’t valuable. I get what you mean by this but when nearly all teams’ “scouting data/systems” is just recording basic data (# of cargo scored, # of panels, climb…), I think validating the raw numbers is far more important/applicable.

#18

Sure, but you can’t necessarily validate all raw numbers from scouting against FRC API data.

Take 2019 for example. Validating climb state is easy. Validating individual robot contribution to the number of scored elements is harder (FRC API only provides alliance totals). Validating cycle times/preferred loading zones/etc. is near impossible.

I think what Caleb is getting at is that there are a lot of things you can be scouting that cannot be derived from the FRC API at all. Therefore, you would expect predictions that come from your scouting data to outperform the best possible prediction algorithm that relies on API data alone.

2 Likes
#19

Ah, I see, yes I believe we were talking about different things. I think you and Eugene summarized the difference pretty well.

When I say this, I’m trying to get at the fundamental idea that all scouting is a way of abstracting robot performance. Every robot has a million different features that make it unique from every other robot. When scouting, you necessarily have to distill all of those differences down to a handful so that you can directly compare teams. Average game pieces scored for example is an excellent metric because it correlates so well with future wins. If you can’t turn the abstraction of your scouting back into successful match predictions, your scouting system is useless, no matter how well you recorded the data. That’s what I mean by becoming dissociated with reality, because you are assigning meaning to numbers that don’t actually have meaning in the real world.

Having reliable raw data is certainly important though, and there are plenty of ways to validate that (API, re-watching videos, comparing your data to other teams’ data, etc…). I can’t think of any other reasonable way to evaluate how meaningful your data are though other than with predictions.

Good discussion though for sure

1 Like
#20

I agree 100% with what you said.

My point is for the average team, I’d be more concerned with validating raw data, for two reasons. First, most teams collect basic match data (number of game pieces scored, climb or not). Secondly, as you stated, average game pieces scored, calculated from that basic match data, is an excellent metric and one that probably doesn’t need to be proven meaningful. If elite teams want to go above and beyond and have advanced stats or formulas, then I agree with validating that specific data to see if it is valuable or meaningful.

I don’t want newer teams thinking they their scouting systems must predict X% of matches right or the system is useless. If you’re collecting basic match data, I think it’s fair to assume that data is meaningful and I’d focus on making sure it’s accurate.

(Kind of related, but I was listening to a podcast that was talking about all the different metrics to quantify QB performance (QBR, QB Rating, PFF Grade, yards per attempt, EPA…) and how poorly QB Rating does at predicting team win-loss compared to the other metrics yet common it is used during tv broadcasts. I think this speaks to @Caleb_Sykes point of how important is it to actually validate that the metrics are actually meaningful and predict future events or if they are just a number that doesn’t really help.)