IYO, whats most important
 

Rank these following critieria from most important, to least important for field scouting, in your humble opinion


Balls Score
Balls Shot
Scoring percentage (Balls Scored/Balls Shot)
Human Player Balls scored
Human Player Balls Shot
Human Player Scoring percentagae
Balls in own trailer
Speed
Traction
Strength
Autonomous Balls Scored
Empty Cell transported
Super Cell Scored
Driving Skill


Feel free to add anything you i might have overlooked. I listed these in no real order

Re: IYO, whats most important
 

a mix of scoring percentage and driver skill/control (the driver could be great, but if the robot doesn't do what its told, its not good)

Re: IYO, whats most important
 

3 Scoring percentage (Balls Scored/Balls Shot)
6 Human Player Scoring percentagae
1 Balls in own trailer
9 Speed
8 Traction
10 Strength
7 Autonomous Balls Scored
5 Empty Cell transported
4 Super Cell Scored
2 Driving Skill

1 = most critical
10 = least critical

Re: IYO, whats most important
 

I think that the most important thing is the difference between the team's score and how much it is scored on, since that is it's overall effect on the match.

Re: IYO, whats most important
 

I think that traction will be the most valuable skill in this competition.

Re: IYO, whats most important
 

Balls Score - Very Important Typically, how many game pieces a robot scores on average is a good indicator if a robot is draftable
Balls Shot - Sort of Important
Scoring percentage (Balls Scored/Balls Shot) - Sort of Important
Human Player Balls scored - Very Important Human player results can differ drastically, and adding their game pieces scored to the robot total should give you a good idea about how the team performs as a whole each match
Human Player Balls Shot - Sort of Important
Human Player Scoring percentagae - Sort of Important
Balls in own trailer - Important This is a good indicator of the evasiveness of the robot and the skill of the driver.
Speed - Not Important Not a field scouting item
Traction - Not Important Not a field scouting item
Strength - Not Important Not a field scouting item
Autonomous Balls Scored - Sort of Important For this game, it is just good to know what your opponents do in autonomous period.
Empty Cell transported - Important You need to know how a robot spends its time during a match, and manipulating empty cells takes away from harvesting and scoring time.
Super Cell Scored - Important However, depending on how the game plays out, the super cell may end up acting just like another moon rock in robots' mechanisms.
Driving Skill - Not Important A much better indicator of driving skill than a subjective field is analyzing the other data fields mentioned above.

Add also:
Team Moon Rocks/Super Cells scored (Robot + Human Player)

Re: IYO, whats most important
 

team compatablity (how well your team gets along with the other)
Driving Skill
ball pick up from floor
Balls Score
Balls Shot
Autonomous Balls Scored
Scoring percentage (Balls Scored/Balls Shot)
Human Player Balls Shot
Human Player Scoring percentagae
Balls in own trailer
Speed
Traction

(in order per me)

Re: IYO, whats most important
 

The one thing that no one ever list in scouting but is one of the most important factors is reliability. I don't care if the robot scores a hundred balls a match if I can't count on you being out there for the next three because it's a fragile as china.

Re: IYO, whats most important
 

While all exterior data should be recorded, especially given the random pairing of opponenets and allies in qualifications, there is ultimately only one stat that matters.

In hockey terms, "+/-". How many points did they score (counting their human player) and how many were scored on them?

Assuming an evened schedule (of not only the "quality" of teams, but the play styles as well), this will show what teams are the best.

Now, given a 7-12 match sample size, it's unlikely that a perfectly balanced schedule will be constructed. So, keep track of who the opponents and allies were, what their play styles were, and how they complimented/detracted eachother.

Look for outliers, and note the factors. Did a bot break down (on either alliance)? Did they have an alliance partner pin the opponent to help them score? Did they get pinned?

Note these outliers to see what play styles will best complimented their abilities, and use that knowledge to your advantage.

Re: IYO, whats most important
 

At the end of the day, these are really the only three that matter:
10 Balls Scored
0 Balls Shot (who cares how many they shoot, so long as they score?)
0 Scoring percentage (Balls Scored/Balls Shot)
0 Human Player Balls scored (this depends on how many balls they are fed)
0 Human Player Balls Shot (depends on how many balls they are fed)
10 Human Player Scoring percentage (you want the human player to score everything they are fed, especially supercells)
10 Balls in own trailer (if they score a lot but are scored on a lot, they are not useful)
0 Speed (if their speed is a problem, it will show in their balls scored and balls-in-own stats)
0 Traction (see comment for speed)
0 Strength (see comment for speed)
5 Autonomous Balls Scored (it might be useful to be able to score in this period, but at the end of the day, its balls scored throughout the match that counts)
5 Empty Cell transported (if they can do it, its handy)
0 Super Cell Scored (covered by HP%)
0 Driving Skill

Reliability is important to note because it might predict failures that could be preventable. If a robot has a preventable failure during a match, it might be a good idea to discount the +/- for that match.

Re: IYO, whats most important
 

I'll separate these out into relevant vs totally non-relevant. Past there, the ranking will be different for each team depending on their strategy. We'll also assume reliability & ability to communicate with their alliance is up there, as those are an 'every year' kind of thing.

Relevant attributes:
Balls Scored
Human Player Balls scored
Empty Cell transported
Traction (gives the ability to push/pin vs other robots)

The rest are irrelevant or are a result of performing something from the list above:
Autonomous Balls Scored (strategy dependent)
Balls in own trailer - this is an iffy one -- it's extremely difficult to control how well your opponents perform, especially with the pinning nature of this game
Balls Shot
Scoring percentage (Balls Scored/Balls Shot)
Human Player Balls Shot
Human Player Scoring percentagae
Speed
Strength
Super Cell Scored
Driving Skill

Re: IYO, whats most important
 

He gets it! :)

All the random other stats (speed, traction, driving skill, etc) will be shown in the stats that actually matter, SCORING!

That being said, they should still be noted to determine play-style and strategy. If you know a team you're going to play in the eliminations is good at pinning other bots, you'll want fast partners who can avoid being pinned. Or if you know an opponent is good at picking off the floor, you'll want bots who don't miss much to reduce their effectiveness. Ultimately, those stats will matter less than the ones that actually change the outcome of the match.

Re: IYO, whats most important
 

It's important to remember that you're scouting with volunteer labor (typically anyways). If you make them record minute details, your scouts will get bored, which will affect their accuracy on the statistics that matter, and may simply rebel and stop recording period. We use mostly parents as round scouts (we're a small team, students are busy keeping the robot running!), and most of them aren't involved much with the game, so we keep it simple.

Re: IYO, whats most important
 

I'll second this. Reliability is one of the most important things we take into account for our scouting. We watch every match a team has, and if they are having a lot of mechanical failures that is often one of the first teams removed from our list.

Re: IYO, whats most important
 

...and yet how do you quantify reliability? If not quantified, how do you assure that you're making non-biased decisions?

Re: IYO, whats most important
 

There are ways.

Let's say that a team is scoring 30-40 points per match, and then suddenly, for a match or two, they go down to 10-20. Then they go back up to 30-40 for a match or two, then down, then up. That's an almost surefire indicator that they are not reliable. It may be that it's different drive teams, or it may be something much more serious. Track me so far?

If a team is consistently getting broken, your scouts can also note that. The team's performance will suffer if they can't fix their robot for the next match.

There are a number of ways to do this, but you can almost certainly do it with the data you already collect.

For me, my biggest concerns this year are: 1: reliability; 2: points scored; 3: points scored on; 4: presence (or not) of an automode. 5 would be collected vs scored; 6, the HP. Other than that, where do they load up?

Re: IYO, whats most important
 

Simple. You create a "range" of scores you deem to be reliable (say, a range of +/-15 points from average). You see how many scores fall within that range of their average score and how many do not. You can then rank teams by reliability.

Re: IYO, whats most important
 

Aka standard deviation. The range of scores from (mean - 1 std. dev) to (mean + 1 std dev.) should hold 75% of all of the data points for a team. The smaller this number, the more consistency there is in the scoring. The great thing about it is that standard deviation is independent of the actual value of the mean (aka average). So if a team consistently scores +/- 15 points from an mean of 20 points per match (i.e. the range is a 5 point match low score to a 35 point match high score), that team will have the same standard deviation as a team who's mean is way up there at 80 who also has a +/- 15 point range.

Then of course, it will be up to the scouts who are watching to eliminate out any outliers from the data. Outliers are matches in which (for example) both of your alliance partners did not work, which is not typical of a match your team is capable of, hence the reason for the very low score.

Re: IYO, whats most important
 

I agree with what Sean is saying.

However, there is/will be a type of robot that does not score, but may produce scoring attempts. The "little box on wheels" bot, that will be able to (try) and pin an opponent in hopes that their teammates can come and score on said opponent.

On a robot like this, speed/traction/driving skill/effectiveness/reliability will all be the MAIN factors, not the irrelevant ones.

Re: IYO, whats most important
 

That's where you look at the "+/-" of the alliance (which can direct equate to score, but might not if you chose to eliminate penalties and/or super cells from the scouting data for this aspect) rather than the individual robot.
While this introduces additional variables (who were their partners), those variables should be able to be defined by the other data you have collected.

For example. Boxbot A is paired with Scorebot A.
If Boxbot A's alliance +/- is considerably higher than usual, it's likely that Scorebot A played a role in that.
If Scorebot A's individual +/- is considerably higher than usual, it's likely that Boxbot A played a role in that.

Reality will be more complex (you'd have to look at how good the opponents are at defense, the 3rd partner, etc), but you get the idea.

Re: IYO, whats most important
 

I see where your going with that, but I still feel like if you put a list down of +/-'s (much like you would do in an alliance captain role) it will be hard to see the value of a robot such as boxbot.

Alliance +/- will reflect this if we assume even alliance distribution. However, me and you both know this often isn't the case, and teams +/-'s may be skewed depending on whether they consistently are with or against good robots.

Re: IYO, whats most important
 

I see where we're getting the disconect. The point of the +/- isn't to be a finished ranking calculation, but rather as data.
It's something you'd review friday night (maybe saturday for procrastinators) to compile your pick lists, not the pick list in itself.

I see as much value in the outliers as I do in the averages. As you review the data, if you notice that time after time scoring teams have good matches when paired with Boxbot A, you might want to pick Boxbot A.

+/- serves as a statistic that will enable quick evaluation of the total data. It will enable you to see how consistent robot's are, as well as quickly identify outlier matches so you can determine what caused them (whether it be a mechanical failure, match-up against strong opponents, or a good partner).

Re: IYO, whats most important
 

Ahh yes...I see where we were not seeing eye to eye now. Yes I agree completely...the outliers have as much value as the averages.

Re: IYO, whats most important
 

Super Cell Scored
Balls Score
Balls in own trailer
Driving Skill
Empty Cell transported
Human Player Balls scored
Human Player Scoring percentage
Traction
Speed
Scoring percentage (Balls Scored/Balls Shot)
Autonomous Balls Scored
Human Player Balls Shot
Strength
Balls Shot

My ranking.

Re: IYO, whats most important
 

my 2 cents;

Points Scored
Pentalities (by the team)
Pentalities (by their Alliance)
Alliance Score (there Alliance)
Oppenents Score (Oppenent's Alliance)
<G14> (y/n)
Super Cell score (1, 2 or 0)

This lets you derive:

Average Score- Both Geometric and Arithmetric go good score comparison.
Average Net Score- Gives a good idea of Team performance.
Percentage of Alliance Score- Lets you see how much they contribute. to their alliance, and with some simple calcutions what they can contribute to yours.
Percentage of <G14>- Lets you gauge if they can really hurt or help.

Re: IYO, whats most important
 

In descending order:

Driving Skill
Speed
Traction
Scoring percentage (Balls Scored/Balls Shot)
Balls Shot
Balls Score
Human Player Balls Shot
Human Player Balls scored
Human Player Scoring percentagae
Autonomous Balls Scored
Strength
Empty Cell transported
Super Cell Scored
Balls in own trailer

Also, it's kind of important to include harvester/shooter movement time...

Re: IYO, whats most important
 

Hmmm ... This give me some ideas.

With this data, could you not tell which robot increased it's partners +/- rating by the most (or highest average increase) ? And wouldn't that show you a robot that may be a great fit for scorers even though itself may not be a prolific scorer?

Re: IYO, whats most important
 

Folks,

Some of the "5th Gear Robotics Simulation" team's visions/goals are the following (stay with me and I will wind up back on topic):

• Improving the current 5th Gear Match simulator to be more of a robot simulator and to allow you to encode the appropriate characteristics from this thread's original post, into the robot models it uses.
• Combine that first improvement with user-customized "AI" algorithms that let users encode different driving/scoring/offense/defense styles/strategies/tactics.
• Combine both of the improvements above with a genetic programming and/or evolving population software "wrapper" around the simulator

The result is a tool that lets you use your opinions about how well your team's robot design (and Human Player abilities) will translate into an actual machine that will accomplish tasks (tasks like avoiding getting pinned, or hanging Rack-N-Roll tubes, or shooting Orbit Balls at a moving target, or...) and then simulate a jillion matches (with/against simulated other robots with simulated abilities you assigned).

At the end of those jillion simulated matches, if you have put in valid assumptions, you will begin to see which designs/strategies (or combinations of designs/strategies in a drafted alliance) are emerging as the ones that can play the game the best (given the assumptions you put into the process). Using genetic programming techniques you can also let designs/strategies evolve away from your original assumptions and see if good ideas emerge from the process.

I promised to finish back on topic - If you were using a simulator to help you turn your opinions into well-founded predictions, at what level of abstraction would you want the simulator to operate? Would you want to feed in parameters like acceleration, speed, and shooting distance/accuracy? or would you want to feed in parameters like # of balls in your own trailer or # of balls successfully shot in autonomous?

In the OP's list, I see multiple levels of abstraction listed, and I'm curious if their is a big preference for operating in the details or at higher levels of abstraction. Operating at the detailed level would probably be more helpful when trying to design a robot; but it is a harder job and requires more work from the users (you).

Blake
PS: Notice that I was careful not to say that a simulator lets you determine the best way to design a robot or play the game. I did try to convey that a decent simulator helps you better understand the implications of your opinions about designs and strategies. The simulator's results will be only as good as the opinions fed into the simulation. Those opinions get replaced with real world physics, with actual human abilties, and other facts, when the rubber hits the road on game day!

Re: IYO, whats most important
 

I believe you have missed one of the most important and easiest to obtain factors... win/loss. While it is significantly influenced by your alliance partners, it is also representative of the total of all other factors. Good driving, good strategy, reliability, HP, scoring ability, etc. It is, after all, how FIRST does it. I wouldn't use win/loss exclusively, but we include it in all of our calculations.

Another point. KISS. You don't want your system dependent on a crew of 12 people making 1-10 rankings on hard-to-judge attributes like driver ability. You get very inconsistant results from person to person. You also burn everyone out if they need to judge 9 attributes of a match.

Our team has historically been very small, never more than 5 students. We have never had more than about 3 people doing scouting. To help make their weekend a little better, and to get more consistent results, I've always recommended a yes/no, high/medium/low evaluation with a multiplier applied to each. This year, we may go with something like:

1) Win/Loss (0, 1) - easiest to get
2) Robot Scoring ability (0, 1, 2) - look for extremes
3) HP Scoring ability (0, 1, 2) - look for extremes
4) Died during match (0, 1) - easy to get
5) Scored heavily on (0, 1) - subjective, but easy to see at the end.

Other more detailed factors are represented by these higher level ones.

Good luck this year, and may you need the results of your scouting efforts.

Re: IYO, whats most important
 

Reading through these posts, I've been playing around with what Sean said - the "+/-" concept and applying it to an individual member on the alliance. In this example - A,B, and C vs. D, E, F.

I would have one scout per team per match. The "A" team scout would record the following:

A-Off score * Number of balls scored in a trailer (including if they have "score" in their own alliance trailer to keep the score competitive and not receive the x2 or x3 penalty). This measures offensive capability.

A-Def score * Number of balls that were scored in their trailer (NOT including if they have balls scored in their trailer by their alliance partner). This measures the defensive capability, how well they avoid being scored on.

A-exc score * Number of Empty Cells exchanged for SuperCells (whether or not they scored the SC, the HP scored the SC, or the SC missed

A-AutoMode * Y/N

A-HP Eff * Low/Med/High - just a general effectiveness rating, rather than a specific shooting percentage. While the HP scoring will play a big part in the alliance scores - tracking that percentage would be a data overload for my scouting team.

So assuming a scenario where the final score is ABC = 40 and DEF = 30, but 10 points were from ABC scoring on its own alliance.

I would calculate a ranking methodology based on the following:
(AOff + BOff + COff) - (DDef + EDef + FDef) = 40
(DOff + EOff + FOff) - (ADef + BDef + CDef) = 20

Then solve via least squares. This is modelled after one of the more effective BCS algorythms called the "Massey Rating." The idea being that solving with this method we take into consideration "strength of schedule."

Let me know what you think and if you have any suggestions to modify this method. I'm planning on trying this out at the DC Regional next week.

Thanks!

Steve

Re: IYO, whats most important
 

Balls Score- important, for pretty obvious reasons
Balls Shot- not as important... if you can shoot 'em but you cant score 'em whats the point?
Scoring percentage (Balls Scored/Balls Shot)- pretty important, see above
Human Player Balls scored
Human Player Balls Shot
Human Player Scoring percentagae- same deal with the robot balls shot
Robot score / Human score- basketball team or robotics team (-;
Balls in own trailer- partially an indicator of how good the driver is.
Speed- you wouldnt drive 60mph on ice, would you?
Traction- extremely important; if you can escape enemies easily then you have a greater chance of winning.
Strength- see woody flowers in the kickoff video.
Autonomous Balls Scored- for the human player or for the robot? a robot that can autonomously locate a target and successfully fire into it is an invaluable asset, especially with turreted bots (the turret control loop, not the driving part), and this will contribute greatly to even the teleop mode
Empty Cell transported- empty cells = super cells. handling them properly is essential.
Super Cell Scored- these make the difference in the games. 14 points behind can turn into 1 point ahead and a victory on einstein.
Driving Skill- insanely important. if your driver can work the skids and the accelerations properly, then you will be able to escape, pin, and score extremely effectively.

Re: IYO, whats most important
 

After attending the Greater Kansas City Regional in week one, we found out this game is incredibly unpredictable. Any team can have a dominant robot, but that is not the most important thing. At the KC regional, as much as 75% of the points scored were scored by the human players. The skill of the human player should be way up there on the list with the most valueable traits.

Something that we realized, again, is that your robot can have a quality design and you will not be as successful as you want. There was a good variety of robots at the KC regional and there was a good mix of robots from top to bottom. The win and loss ranking that FIRST is using this year can really hurt some teams. If a team gets a "bad" alliance pairing, a robot doesn't show up, or a robot breaks down, it is pretty much over with.

In my opinion, you want to have a great robot that can do it all, but with what we've seen in week one, is that the human player has the biggest impact on the game. I like having a team that can have total control of their robot, maneuvering their way around and avoid being scored on. Robot scoring just wasn't all I had expected this year.

Re: IYO, whats most important
 

The entire goal of scouting is to put together a good ALLIANCE if you make it to a top 8 spot. The best ALLIANCES are not necessarily made up of the three best teams as determined by points scored or points against or any other stat. The best ALLIANCES are three teams whose strengths play to each other, can work together, and complement each other's skills. If your bot is amazing at pinning, you need a bot that can power dump on your side, even if you have to reach down a bit on your stat sheet. Therefore, you should separate your stat sheet into categories based on primary objectives of that bot in any given match:
Defense/Pinning
Dumpers/Shooters
Empty Cell runners
and maybe a general category for reliability, consistency, and versatility

Rank your field in these categories, then look for what you need to create a dynamic alliance.

Lastly, DONT pick your alliance simply by rank on the leaderboard, especially in this game.

Re: IYO, whats most important
 

For scouting teams should keep a tally of how many penalties other teams Payload Specialist accumulate.

Re: IYO, whats most important
 

In my opinion for scouting it should be:

Balls Scored (because it doesn't matter about how many it shot, rather how many can it score; in a case where a robot has 100% accuracy but can only score once or twice with a few balls it doesn't matter about percentage)

Human Player Percentage and Score( for this one percentage does matter, if
a HP has 20 moon rocks and makes 18, that is very good. However, if a HP somehow obtains 30 or 40 -by robots herding- and makes 18 then that is not that good compared to the previous HP)

Balls in own trailer (does not matter much because in some cases if a robot is pinning another, they may be scored on more so than one that is not doing anything but running away)

Strength (this is included with traction because the only way strength is used in this game is by pinning another robot or pushing them away from scoring)

Empty Cell transported (may be helpful but not that important)

Super Cell Scored (It also depends if they had any empty cells and if the robots were near them; it should go under the Human Player Percentage)

Driving Skill (this includes speed, because what use is it to have a speedy robot that cant do anything but run into walls? Driving skill should be a combination of how well can they control the robot and how fast can they drive the robot)

Reliability(as many other people mentioned, is this team consistently playing this well)

The scouts should also make the lists into 3 groups
1. Offensive (Dumping, Shooting etc)
2. Defensive (Blocking, pinning, distracting etc)
3. Empty Cell transfer