The merits of treating robotics tourneys like a game of Fire Emblem

[Amy33Amy33]

So at 10K lakes, we had an argument between a mentor and the scouting captain over the methodology used to scout. The mentor obviously won, as using time tested methodology is safer. But I want to talk about the methodology of the scouting captain.

See, our team has struggled with scouting a lot over the years, so this year we elected a freshman as our scouting captain to breath fresh life into our scouting system given that we're a fairly small team and we never have enough people to run a full six person scouting team, so we usually end up trying to run a six person scouting team built for six people with two people. The mentor was right in that at an actual competition, running a scouting team for six with two people, especially in the rare year that our robot didn't break much at all and we actually had six people to run a scouting team (YES!) (Well we did, until our electrical was ripped out of the robot and we needed to rewire the whole robot on the fly which was no bueno for the scouting team which was basically the electrical team with pencils...)

But I'm not here to talk about the methodology of the mentor, we all know what a six person scouting team is and how it works.

I'm here to talk about the methodology of the student.

See, the student is a bit of a Fire Emblem addict.

Since I was elected marketing captain, and since I was a junior, I had a 3DS gathering dust on a table, and after I broke up with my boyfriend and he never wanted to be friends with me ever again (he was my gaming buddy, and the person who had introduced me to gaming when I was in elementary school) I basically quit gaming, it was genuinely a useless waste of time. Except, I didn't really want to sell my 3DS, I still pulled out Pokemon Mystery Dungeon once in a while, I was like that old person who owns a sewing machine but doesn't use it, but doesn't want to get rid of it. So when my buddy the scouting captain was like "I want to play fire emblem for the new 3DS but I'm a high school student on a robotics team and I'm poor" I was like "Bruh, here, you can use it for a while"

Hasn't given it back yet...

Anyways, here's how it works. He sits there and watches the robots and talks. All the robots. His assistant sits next to him, and takes all the notes. The person taking notes switches out periodically to tell the drive team what's been going on.

Part of the reason that he lost the argument was personal, the guy missed the practice matches on the first day to take a test, though I'm not sure if he needed to be there, our team doesn't scout on the first day except for it does all the pit scouting on the first day. The other part of the reason was that we actually got a printer and printed out the notes for the mentor, and the notes were basically a log of

Match 1, R22 - B94, R: 1Robot, 2Robot, 3Robot, B: 4Robot, 5Robot, 6Robot

1Robot is terrible
2Robot is terrible
3Robot is terrible

With an occasional comment of

4Robot mediocre, do not let 4Robot use autonomous

and

5Robot great with feeder.

Not very useful if you're a mentor, and definitely not the sort of thing marketing would want leaked onto the internet. But the list actually makes sense if you think about it the way that he was thinking about it. The notes aren't strategy notes. They're a battle log, each of the robots are characters. The game, Fire Emblem. His strategy for scouting relied on treating robots like characters in Fire Emblem, treating their actions like weapons, and treating the totes as enemies on a map, and treating the game itself like a weird map in Fire Emblem. One problem though. If you are judging characters in Fire Emblem on strengths and weaknesses, you can't do so unless you have character death. Ricken dies alot, so I wont use him. When Robin uses swords, he dies more often. If you treat robots breaking down as death, you aren't going to be able to judge them effectively. A thing that you will notice very quickly if you are an outsider reading his notes is that every robot except for the top three or so are terrible, but also that his notes are in chronological order, top being the beginning of the match, bottom being the end.

Match 1, R22 - B94, R: 1Robot, 2Robot, 3Robot, B: 4Robot, 5Robot, 6Robot

1Robot is terrible, tipped during auto
4Robot is mediocre, do not let 4Robot use autonomous
6Robot = slow reaction time, is terrible
2Robot is terrible
3Robot is terrible
4Robot is terrible
5Robot great with feeder.

Basically, he built character death into his system to judge robots more effectively. By judging chronologically where the robots are terrible, you can create a rank list, building experience into the system by ranking robots against eachother, and not ranking the outcomes of match one against the outcomes of match 64, but by ranking the outcomes against each other, and then combining them to form a ranking, using match scores to break ties.

Match 30, R:50 - B: 90

7Robot is terrible, crashed into 8Robot
9Robot is terrible
10Robot is terrible
6Robot is terrible
8Robot is good on toes
11Robot is great on feeder

Rank list. Bottom = Want
1Robot
7Robot
9Robot
10Robot
4Robot
6Robot
2Robot
3Robot
4Robot
8Robot
5Robot
11Robot

Of course, this list either requires pit scouting data to interpret or someone who has memorized all the robots, but oddly enough our scouting captain pretty much did memorize all the robots, and I'm sure we all have a pretty good idea of how he did it...

http://i59.tinypic.com/20ro4l1.png

All that's left to do is refine the system, and get some actual criteria to define death, and it should actually work well. Maybe add in a third person to compile and run information to the drive team

[John Retkowski]

You had me at Fire Emblem, but at the end of the day does this system help your teams alliance selection representative when they're out on the field with a thousand people staring at them? For me, scouting is making a list of what robots pair up best with you, and what robots pair up best with your opponents. Of course things like consistency(especially for this year), driver skill, and breaking often or not breaking often are also extremely important. This system seems like a glorified version of the event ranking system with fun facts about each robot added in.

And by the way, Robin's weapons depend on what stat bonus you give him/her at the beginning of the game, and what class he/she is. As a grandmaster magic might be better, but as a dread fighter, swords and axes ftw.

I wish I had time to play fire emblem again...

[Jay O'Donnell]

So basically the difference between the two systems (from what I can tell) is that the original system I assume uses quantitative data (how many stacks they made or points they scored, or whatever metric you used) and the students method was qualitative.

Qualitative data is insanely risky. Using words like great or good isn't an effective method of keeping data. What does good mean? What does great mean? How do you determine if one great is better than another great? If the reasoning for using the student's method is that it shows how a robot improves or not, why not just use the quantitative data (points scored) to do the same thing? Just keep a sheet (or excel column) for each team and put down how many points they score each match and you get the same effect as the student's method, but with a better way to compare the teams.

You're also throwing all of your scouting eggs into one basket. What if the kid forgets a good team? What if he can't show up one day and he doesn't know beforehand? If your entire scouting system depends on one kid being there, you're not only risking it's failure if he can't be there, but not using a system anyone else can take over. Not to mention it's not a system you'll likely be able to use after he graduates.

When I scouted my freshman year we had basically the same setup. I memorized all of the teams, and myself and a senior were the only scouts. We thought we made great picks at the time, but looking back on it 3 years later they were just based off our opinions, and not any real data. Because of that we didn't pick the right teams.

TLDR: I'm not saying that the student's system can't work, but it's definitely not an optimal setup. Quantitative data>Qualitative data.

Just my $.02

[evanperryg]

Originally Posted by Jay O'Donnell

Quantitative data>Qualitative data.

There was a rather heated thread a little while ago about this, but that isn't necessarily true. Quantitative data without qualitative is too objective to form a well-rounded picklist, just as qualitative without quantitative is to subjective to form a well-rounded picklist. Often times, it's important to know how or why a team is doing something. These things can be vital in ensuring the picklist takes into account teams who have underperformed due to alliances, and teams who got lucky a few times. Qualitative data also ensures you know teams that no-showed, didn't move during a match, or had a major failure on the field. This sort of info could be taken alongside quantitative, but it's easier to do it qualitatively as it takes little effort while reducing the amount of work quantitative scouters need to do.

That said, if that's what OP's scouter's notes look like, he's doing it wrong. Data that just says "bad" doesn't really mean anything. Good qualitative data should explain how a team is doing something, their strengths and weaknesses, a unique, defining trait of the robot, etc. If that's what this scouter is doing, then odds are your aggregated info for most of the teams would look a little like this:
"bad bad bad bad bad bad bad bad bad bad bad"
which is hardly helpful at all.

[Kevin Leonard]

I'm not sure what to make of this. It sounds pretty absurd, especially if the bottom of your rankings ended up looking like "Team XXXX Terrible, Team XXXX Terrible"
How does this sort through the bottom of the picklist and identify teams with desirable qualities?

If using this system and Fire Emblem works for you, go for it. I'd love to just see an example picklist or document showing how this system worked for you at an event just for curiosity's sake. However, I don't want some rookie team trying to figure out how to scout looking on Chief Delphi for the first time and seeing this and using it as a model.

This also relies heavily on one individual's judgement, whereas I much prefer to make scouting and strategy decisions based on the consensus and debate of multiple intelligent, knowledgeable individuals.

I've seen very, very smart people with enormous amounts of knowledge make stupid decisions about scouting based on a silly notion stuck in their head.

[Jon Stratis]

When we scout, we look for quantitative data, not qualitative. Having notes that "robot1 is terrible" is qualitative - there's no basis for comparison between two robots that are considered terrible. Quantitative data, on the other hand, would tell you the final scores, how many totes each robot stacked, etc - Room for a direct comparison between robots, as even "terrible" ones would have two numbers that could be compared.

If you can, I recommend you guys sign up and attend SRS in two weeks (Since your a local MN team!) on Saturday the 18th. I believe 2052 is doing a session there on scouting which could be valuable, and I'm doing a session that includes a lot of details about the FRC Event API, a service FIRST created last fall that lets you get information from the FMS about every match played - it could definitely be used as part of a scouting program, written before the event, that could give you some nice rankings of teams based on quantitative data!. And the best part - you don't need 6 people watching every match to use it, you can just run your program at the end of the day and get info for every match played!

[jajabinx124]

Originally Posted by evanperryg

Good qualitative data should explain how a team is doing something, their strengths and weaknesses, a unique, defining trait of the robot, etc. If that's what this scouter is doing, then odds are your aggregated info for most of the teams would look a little like this:
"bad bad bad bad bad bad bad bad bad bad bad"
which is hardly helpful at all.

Agreed. Watching a match and just saying that robot played terribly or bad isn't gonna be helpful. If you want a basis for comparison between 2 or more robots learn how summarize the match for the robot you are scouting(It's strengths, weaknesses, traits, etc. that you observed).

Here is an example of good qualitative data,

Team XXXX, Match XX

Auto: Attempted 3 tote stack in auto but failed
Teleop: Did coop, was fast at creating 2 non-capped stacks of 3 from the landfill area, stopped randomly and didn't move for the rest of the match

Our team has scouted quantitatively in the past, but recently we've been telling some of our more experienced scouts to scout qualitatively as well because we find qualitative data to be useful during strategy meetings(we find them useful because it provides another basis of comparison between teams after comparing teams regarding how they performed quantitatively, overall it's another topic to discuss about during our strategy meeting which makes our picklists stronger).

[IKE]

Single person scouting can be effective, but I think the organization could use a little work.

Several years ago, I did a subjective scouting booklet loosely based off of some "tasting" booklets I had observed online. The intent was to fill out each page with some sort of full subjective assessment for each team.
http://www.chiefdelphi.com/media/papers/2572

When using this, I would typically use it with another sheet, and a copy of the schedule to help ensure that I got at least 1 full matches worth of each team. One weakness of this is that you can closely watch a teams 1 bad match, and possibly miss several good one thus classifying what would be a "good" team as a weak time. Vice Versa is also possible.

The system I mentioned above does tend to work pretty well for match scouting used to influence match strategy where you watch your partners and opponents 1-2 rounds ahead of your match with/against them, and use the comments to feed a strategy card.

***************************
In contrast though, I like the concept of the type of scouting you are discussing.
Let's say I just gave a 5-1 (use a 0 for no-show) rating for every robot based off of their performance. I could in general watch the match from a wholistic viewpoint, give a performance rating for every robot, for every match. The average and variances of rating would likely give me rough results that would mirror a lot of pick-lists that use some sort of performance based system ranking.
It wouldn't be as accurate, but it if using 2 people (one to scout, and one to take notes), it is about 1/8th the effort, and is likely better than 12.5% accuracy that the portion of work would say is equitable. My guess is if you were to figure out a rating method and compare it to others lists, your list would probably be around 75-80% as good as theirs. I know this is possible, because without using a "formal" system, I am usually able to get about 20/24 teams playing in elims at District events just watching matches as an observer. Often I am on the order of 15/16 for the first 16 teams with greater misses on the back end.

With a little practice, and 1-5 or 1-6 (again 0 for no shows), you could likely get a pretty good list.
If you had a group of about 4 students, and you did Red/Blue Defense/Offense ratings, you could likely get really good results as long as your Red/Blue teams discussed and synchronized their ratings (IE an get concurrence of what a 3, 4, and 5 look like).

If you took this data for all matches, you could likely see the important trends that could take your list from a 75-80% accurate level to a 90%+

You still won't be as accurate as the teams collecting stats, that have the ability to process the stats and refine the value of those stats. I would venture though, that your list may be more accurate than the teams that collect wonderful stats, but do not have a clear vision of how to use them....

[logank013]

Quantitative data i always better. This year, we did some turning qualitative data into quantitative data. For instance, instead of say "good stacker, takes a while with cans" We would used a stopwatch and time how long it took to make a full stack. how long of that was getting the can? how long was it to get 6 totes stacked under the can" So we would use a stopwatch. What we did was we took most of our quantitative data on the first day. Then, we made a list of robots for the second day that were "watch for these guys" then, we would do some qualitative data. Now "bad with cans" is qualitative. We would take it a step further and say "It takes 20 seconds to get the can and drive back to the human player station" Then, we could turn it into quantitative data and use averages to make some small part in the decision. At our cori off season event, we took a small team and i was the only scouter to scout 6 robots. for this game, it was fairly easy. I used excel and I had a notes section and i had a stacks section. So i would say "2 full stacks" in the stacks section and i would take some side notes. This would help me turn it into quantitative data since i knew that that robot was capable of putting up 84 points. You can still have quantitative data with a small number of scouters. Just find a quick way to jot down notes that can be turned into quantitative data later for excel or however you calculate quantitative data. Hoefully, this made some sense.

[Abhishek R]

Like others have said, this basically boils down to quantitative (your mentor) vs subjective (the student) data. Like most things in life, the answer is usually some sort of happy median. The fact that you have someone taking such detailed notes on every team already is a great start.

Depending on who's doing the scouting and how, I think there's opportunity for subjective to be better than quantitative - statistics can't tell you everything.

[gblake]

If you ask the right scouting questions, you will (can) get the right answers.

[logank013]

Originally Posted by gblake

If you ask the right scouting questions, you will (can) get the right answers.

Yeah. Make sure your scouting data is good too. We had some odd things in ours. Like a team "got 8 cans from the step in autonomous" and a "team knocked over 10 stacks" just random weirdness so mark sure that is edited out.

[The other Gabe]

my co- leader and I do something very similar to this, except we write on notecards (1 for each team), and each of us takes one of the alliances. these notecards become my memory bank- when I need to know if a robot is decent or not, I look them up. We have the benefit, however, of having a 6 person scouting team as well (well kinda, more like off- duty pit members that sometimes dont show up, and one or two dedicated to scouting guys that never get breaks because of them ), so we can use the data in combination with the cards for an even greater effect. that being said, if you have a small team, that's a brilliant way to run scouting, especially if your lead really knows the robots well.

[EricH]

Sounds like you've got your "overall" scout: one who can look over the match and get a general picture of who does what and how well.


The problem is, you still need "individual robot" scouts. While the electrical team who isn't doing anything may be useful while they're there, I suggest taking a less temporary/less subject to the demands of the robot approach and "impressing" (early 1800s British sense) anybody who isn't a) part of the drive team or b) part of the pit crew at the time (which will ideally be as few as possible, but NOT the drive team).

And by "anybody" I mean "anybody". Anybody that is in any way affiliated with the team should be considered fair game for "recruitment"--this includes mentors, students of all ages and grades, parents, siblings that drop by, alumni...

And, ideally, those individual robot scouts will also be taking down some subjective notes.

[CJ_Elliott]

525, 2502, 3184... all deez tems iz bad. 1711, 1711, 1711... iz gud allyances