Merry Christmas!
Quote:
Originally Posted by Joe Ross
How did you validate the accuracy of your scouting data?
|
At our events, we scouted collaboratively with other teams. Accuracy actually became a prime concern of ours, so we had every entry into our scouting database checked over by our head scout before entry (who had been watching each match as a whole immediately before validating). Additionally, we recorded matches for further verification. There were a few instances I can recall where errors were detected in entries, and we used our footage to re-scout that robot for that match.
Quote:
Originally Posted by Joe Ross
Why did you choose to use percent error, rather then absolute error?
|
This was an attempt to make the results of our paper more applicable to other events in the 2013 season. Archimedes, for instance, is hardly indicative of the average regional or district event, and yet it forms more than half of our data (51%, to be precise; out of 196 event/team combinations sampled, 100 were from Archimedes).
Quote:
Originally Posted by Joe Ross
Do the results change if you only look at the top 24 or 30 teams at an event (the teams you would be considering when forming a pick list)?
Is there an OPR at which it becomes more accurate? Looking at chart 13, OPRs above 15 seem much better then those below 15 (obviously game dependent).
Can you quantify the percent chance that Team A is better then Team B, given a specific OPR difference (IE, Team A has an OPR that is 1 higher then Team B, and has a 55% chance of being better then Team B, but Team C has an OPR that is 10 higher then Team B, and has a 90% chance of being better then Team B.
Does the percent error histogram still look normal if you discard the outliers and put more bins between -100% and +200%?
|
Let me get back to you on answering your questions backed up with relevant diagrams and calculations. I'd also like to look at posting our dataset too. However, here are my suspicions:
- If you see my response Basel A's question below, sampling only the top 30 or so teams from each of our competitions should decrease the percent error, which would change the percent/tolerance table for the better. OPR should become more accurate.
- I have no idea. Really intriguing question, though.
- Since we do have the averages and standard deviations for each team, you could let Team A and B be represented by a variable with mu equal to their Average Score and sigma equal to the Standard Deviation in their score. If you subtract the two means and add the two variances, you should be able to find the number you're looking for by integrating from -infinity to 0 and 0 to infinity. We used this method to predict match outcomes, but it was not very accurate. I don't know of a simple way to extend that to OPR, though.
- I'll get back to you.
Quote:
Originally Posted by Basel A
These are all good things to think about. Particularly, you may want to reconsider your use of percent error as opposed to absolute error. I'm estimating here, but it looks like absolute error would've been pretty consistent regardless of true scoring. Take a look at the correlation of those; I'd bet very little of the variation in absolute error is explained by variation in true scoring average.
|
If you look at the scatterplot on slide 9 and compare the least squares line to the scatterplot, you'll see that absolute error decreases as Average Points Scored increases (I did have a residuals plot in here previously, but it looks like I accidentally removed it before I posted it). The percent error decreasing as Average Points Scored is not simply a function of the denominator for the percent error calculation increasing.
Quote:
Originally Posted by Basel A
If there were such a correlation, you would have noticed it in the residual plot for your OPR-True Average linear regression. I didn't notice a residual plot in your pdf; they are essential for determining if your model is a good fit for the data.
|
I did do a residual plot, and include it previously; it must have been accidentally removed during my revising. I'll make sure to fix that to back up my previous claim (see answer to previous question).