IYO, whats most important

[EricH]

Originally Posted by SentientCitrus

...and yet how do you quantify reliability? If not quantified, how do you assure that you're making non-biased decisions?

There are ways.

Let's say that a team is scoring 30-40 points per match, and then suddenly, for a match or two, they go down to 10-20. Then they go back up to 30-40 for a match or two, then down, then up. That's an almost surefire indicator that they are not reliable. It may be that it's different drive teams, or it may be something much more serious. Track me so far?

If a team is consistently getting broken, your scouts can also note that. The team's performance will suffer if they can't fix their robot for the next match.

There are a number of ways to do this, but you can almost certainly do it with the data you already collect.

For me, my biggest concerns this year are: 1: reliability; 2: points scored; 3: points scored on; 4: presence (or not) of an automode. 5 would be collected vs scored; 6, the HP. Other than that, where do they load up?

[Lil' Lavery]

Originally Posted by SentientCitrus

...and yet how do you quantify reliability? If not quantified, how do you assure that you're making non-biased decisions?

Simple. You create a "range" of scores you deem to be reliable (say, a range of +/-15 points from average). You see how many scores fall within that range of their average score and how many do not. You can then rank teams by reliability.

[JesseK]

Originally Posted by Lil' Lavery

Simple. You create a "range" of scores you deem to be reliable (say, a range of +/-15 points from average). You see how many scores fall within that range of their average score and how many do not. You can then rank teams by reliability.

Aka standard deviation. The range of scores from (mean - 1 std. dev) to (mean + 1 std dev.) should hold 75% of all of the data points for a team. The smaller this number, the more consistency there is in the scoring. The great thing about it is that standard deviation is independent of the actual value of the mean (aka average). So if a team consistently scores +/- 15 points from an mean of 20 points per match (i.e. the range is a 5 point match low score to a 35 point match high score), that team will have the same standard deviation as a team who's mean is way up there at 80 who also has a +/- 15 point range.

Then of course, it will be up to the scouts who are watching to eliminate out any outliers from the data. Outliers are matches in which (for example) both of your alliance partners did not work, which is not typical of a match your team is capable of, hence the reason for the very low score.

[Brandon Holley]

I agree with what Sean is saying.

However, there is/will be a type of robot that does not score, but may produce scoring attempts. The "little box on wheels" bot, that will be able to (try) and pin an opponent in hopes that their teammates can come and score on said opponent.

On a robot like this, speed/traction/driving skill/effectiveness/reliability will all be the MAIN factors, not the irrelevant ones.

[Lil' Lavery]

Originally Posted by Brandon Holley

I agree with what Sean is saying.

However, there is/will be a type of robot that does not score, but may produce scoring attempts. The "little box on wheels" bot, that will be able to (try) and pin an opponent in hopes that their teammates can come and score on said opponent.

On a robot like this, speed/traction/driving skill/effectiveness/reliability will all be the MAIN factors, not the irrelevant ones.

That's where you look at the "+/-" of the alliance (which can direct equate to score, but might not if you chose to eliminate penalties and/or super cells from the scouting data for this aspect) rather than the individual robot.
While this introduces additional variables (who were their partners), those variables should be able to be defined by the other data you have collected.

For example. Boxbot A is paired with Scorebot A.
If Boxbot A's alliance +/- is considerably higher than usual, it's likely that Scorebot A played a role in that.
If Scorebot A's individual +/- is considerably higher than usual, it's likely that Boxbot A played a role in that.

Reality will be more complex (you'd have to look at how good the opponents are at defense, the 3rd partner, etc), but you get the idea.

[Brandon Holley]

Originally Posted by Lil' Lavery

For example. Boxbot A is paired with Scorebot A.
If Boxbot A's alliance +/- is considerably higher than usual, it's likely that Scorebot A played a role in that.
If Scorebot A's individual +/- is considerably higher than usual, it's likely that Boxbot A played a role in that.

Reality will be more complex (you'd have to look at how good the opponents are at defense, the 3rd partner, etc), but you get the idea.

I see where your going with that, but I still feel like if you put a list down of +/-'s (much like you would do in an alliance captain role) it will be hard to see the value of a robot such as boxbot.

Alliance +/- will reflect this if we assume even alliance distribution. However, me and you both know this often isn't the case, and teams +/-'s may be skewed depending on whether they consistently are with or against good robots.

[Lil' Lavery]

Originally Posted by Brandon Holley

I see where your going with that, but I still feel like if you put a list down of +/-'s (much like you would do in an alliance captain role) it will be hard to see the value of a robot such as boxbot.

Alliance +/- will reflect this if we assume even alliance distribution. However, me and you both know this often isn't the case, and teams +/-'s may be skewed depending on whether they consistently are with or against good robots.

I see where we're getting the disconect. The point of the +/- isn't to be a finished ranking calculation, but rather as data.
It's something you'd review friday night (maybe saturday for procrastinators) to compile your pick lists, not the pick list in itself.

I see as much value in the outliers as I do in the averages. As you review the data, if you notice that time after time scoring teams have good matches when paired with Boxbot A, you might want to pick Boxbot A.

+/- serves as a statistic that will enable quick evaluation of the total data. It will enable you to see how consistent robot's are, as well as quickly identify outlier matches so you can determine what caused them (whether it be a mechanical failure, match-up against strong opponents, or a good partner).

[Brandon Holley]

Originally Posted by Lil' Lavery

I see where we're getting the disconect. The point of the +/- isn't to be a finished ranking calculation, but rather as data.
It's something you'd review friday night (maybe saturday for procrastinators) to compile your pick lists, not the pick list in itself.

I see as much value in the outliers as I do in the averages. As you review the data, if you notice that time after time scoring teams have good matches when paired with Boxbot A, you might want to pick Boxbot A.

+/- serves as a statistic that will enable quick evaluation of the total data. It will enable you to see how consistent robot's are, as well as quickly identify outlier matches so you can determine what caused them (whether it be a mechanical failure, match-up against strong opponents, or a good partner).

Ahh yes...I see where we were not seeing eye to eye now. Yes I agree completely...the outliers have as much value as the averages.

[Woody1458]

Super Cell Scored
Balls Score
Balls in own trailer
Driving Skill
Empty Cell transported
Human Player Balls scored
Human Player Scoring percentage
Traction
Speed
Scoring percentage (Balls Scored/Balls Shot)
Autonomous Balls Scored
Human Player Balls Shot
Strength
Balls Shot

My ranking.

[XXShadowXX]

my 2 cents;

Points Scored
Pentalities (by the team)
Pentalities (by their Alliance)
Alliance Score (there Alliance)
Oppenents Score (Oppenent's Alliance)
<G14> (y/n)
Super Cell score (1, 2 or 0)

This lets you derive:

Average Score- Both Geometric and Arithmetric go good score comparison.
Average Net Score- Gives a good idea of Team performance.
Percentage of Alliance Score- Lets you see how much they contribute. to their alliance, and with some simple calcutions what they can contribute to yours.
Percentage of <G14>- Lets you gauge if they can really hurt or help.

[danderson]

In descending order:

Driving Skill
Speed
Traction
Scoring percentage (Balls Scored/Balls Shot)
Balls Shot
Balls Score
Human Player Balls Shot
Human Player Balls scored
Human Player Scoring percentagae
Autonomous Balls Scored
Strength
Empty Cell transported
Super Cell Scored
Balls in own trailer

Also, it's kind of important to include harvester/shooter movement time...

[Daniel_LaFleur]

Originally Posted by Lil' Lavery

I see where we're getting the disconect. The point of the +/- isn't to be a finished ranking calculation, but rather as data.
It's something you'd review friday night (maybe saturday for procrastinators) to compile your pick lists, not the pick list in itself.

I see as much value in the outliers as I do in the averages. As you review the data, if you notice that time after time scoring teams have good matches when paired with Boxbot A, you might want to pick Boxbot A.

+/- serves as a statistic that will enable quick evaluation of the total data. It will enable you to see how consistent robot's are, as well as quickly identify outlier matches so you can determine what caused them (whether it be a mechanical failure, match-up against strong opponents, or a good partner).

Hmmm ... This give me some ideas.

With this data, could you not tell which robot increased it's partners +/- rating by the most (or highest average increase) ? And wouldn't that show you a robot that may be a great fit for scorers even though itself may not be a prolific scorer?

[gblake]

Originally Posted by Dantvman27

Rank these following criteria from most important, to least important for field scouting, in your humble opinion


Balls Score
Balls Shot
Scoring percentage (Balls Scored/Balls Shot)
Human Player Balls scored
Human Player Balls Shot
Human Player Scoring percentage
Balls in own trailer
Speed
Traction
Strength
Autonomous Balls Scored
Empty Cell transported
Super Cell Scored
Driving Skill


Feel free to add anything you i might have overlooked. I listed these in no real order

Folks,

Some of the "5th Gear Robotics Simulation" team's visions/goals are the following (stay with me and I will wind up back on topic):

• Improving the current 5th Gear Match simulator to be more of a robot simulator and to allow you to encode the appropriate characteristics from this thread's original post, into the robot models it uses.
• Combine that first improvement with user-customized "AI" algorithms that let users encode different driving/scoring/offense/defense styles/strategies/tactics.
• Combine both of the improvements above with a genetic programming and/or evolving population software "wrapper" around the simulator

The result is a tool that lets you use your opinions about how well your team's robot design (and Human Player abilities) will translate into an actual machine that will accomplish tasks (tasks like avoiding getting pinned, or hanging Rack-N-Roll tubes, or shooting Orbit Balls at a moving target, or...) and then simulate a jillion matches (with/against simulated other robots with simulated abilities you assigned).

At the end of those jillion simulated matches, if you have put in valid assumptions, you will begin to see which designs/strategies (or combinations of designs/strategies in a drafted alliance) are emerging as the ones that can play the game the best (given the assumptions you put into the process). Using genetic programming techniques you can also let designs/strategies evolve away from your original assumptions and see if good ideas emerge from the process.

I promised to finish back on topic - If you were using a simulator to help you turn your opinions into well-founded predictions, at what level of abstraction would you want the simulator to operate? Would you want to feed in parameters like acceleration, speed, and shooting distance/accuracy? or would you want to feed in parameters like # of balls in your own trailer or # of balls successfully shot in autonomous?

In the OP's list, I see multiple levels of abstraction listed, and I'm curious if their is a big preference for operating in the details or at higher levels of abstraction. Operating at the detailed level would probably be more helpful when trying to design a robot; but it is a harder job and requires more work from the users (you).

Blake
PS: Notice that I was careful not to say that a simulator lets you determine the best way to design a robot or play the game. I did try to convey that a decent simulator helps you better understand the implications of your opinions about designs and strategies. The simulator's results will be only as good as the opinions fed into the simulation. Those opinions get replaced with real world physics, with actual human abilties, and other facts, when the rubber hits the road on game day!

[theCoder]

I believe you have missed one of the most important and easiest to obtain factors... win/loss. While it is significantly influenced by your alliance partners, it is also representative of the total of all other factors. Good driving, good strategy, reliability, HP, scoring ability, etc. It is, after all, how FIRST does it. I wouldn't use win/loss exclusively, but we include it in all of our calculations.

Another point. KISS. You don't want your system dependent on a crew of 12 people making 1-10 rankings on hard-to-judge attributes like driver ability. You get very inconsistant results from person to person. You also burn everyone out if they need to judge 9 attributes of a match.

Our team has historically been very small, never more than 5 students. We have never had more than about 3 people doing scouting. To help make their weekend a little better, and to get more consistent results, I've always recommended a yes/no, high/medium/low evaluation with a multiplier applied to each. This year, we may go with something like:

1) Win/Loss (0, 1) - easiest to get
2) Robot Scoring ability (0, 1, 2) - look for extremes
3) HP Scoring ability (0, 1, 2) - look for extremes
4) Died during match (0, 1) - easy to get
5) Scored heavily on (0, 1) - subjective, but easy to see at the end.

Other more detailed factors are represented by these higher level ones.

Good luck this year, and may you need the results of your scouting efforts.

[SteveGPage]

Reading through these posts, I've been playing around with what Sean said - the "+/-" concept and applying it to an individual member on the alliance. In this example - A,B, and C vs. D, E, F.

I would have one scout per team per match. The "A" team scout would record the following:

A-Off score * Number of balls scored in a trailer (including if they have "score" in their own alliance trailer to keep the score competitive and not receive the x2 or x3 penalty). This measures offensive capability.

A-Def score * Number of balls that were scored in their trailer (NOT including if they have balls scored in their trailer by their alliance partner). This measures the defensive capability, how well they avoid being scored on.

A-exc score * Number of Empty Cells exchanged for SuperCells (whether or not they scored the SC, the HP scored the SC, or the SC missed

A-AutoMode * Y/N

A-HP Eff * Low/Med/High - just a general effectiveness rating, rather than a specific shooting percentage. While the HP scoring will play a big part in the alliance scores - tracking that percentage would be a data overload for my scouting team.

So assuming a scenario where the final score is ABC = 40 and DEF = 30, but 10 points were from ABC scoring on its own alliance.

I would calculate a ranking methodology based on the following:
(AOff + BOff + COff) - (DDef + EDef + FDef) = 40
(DOff + EOff + FOff) - (ADef + BDef + CDef) = 20

Then solve via least squares. This is modelled after one of the more effective BCS algorythms called the "Massey Rating." The idea being that solving with this method we take into consideration "strength of schedule."

Let me know what you think and if you have any suggestions to modify this method. I'm planning on trying this out at the DC Regional next week.

Thanks!

Steve