Requirements- What Data to Collect?

Well, this is the first of what will be a series of threads to determine the necessary requirements for a viable FSN system. Now that I’ve thrown out a bunch of legal-sounding garbage, let me say this in plain English. We want to know what you want FSN to do. These “Requirements” threads will attempt to figure that out.

The topic I chose to start with is a fairly big one: data collection. My question to the FIRST scouting community is this: what data should we collect, and how should we gather it? How can we best determine what data fields should be collected and submitted? Who will choose what is the “official standard,” so to speak? If you have anything to say regarding the actual scouting data that should be collected, this is the place to do it!

Note: This is not about methods of collection- just the data itself. See my second question for collection methods.

A lot of this depends on what the game is next year. There are lots of game specific things such as stacking ability/speed for this year. These fields can be discussed here during the build season, then finally decided by a popular vote or some such thing.

One thing you can guarantee will be necessary is drive train. For preliminary scouting, fields for motors used, gear ratio, and shifting ability would be perfect. In match evaluation is a little harder to do objectively. A time to the top of the ramp(or something equivalent to fit with next year’s game) could be used for speed, but then you’d have to keep in mind that some teams have their autonomous run at a slower speed. A subjective rating(1-5) could be used for both, but that poses problems with different people having different standards/ratings. Earlier Erin suggested using a system where the mode is displayed on the main page and the percentages are shown on another page. This could help eliminate the subjectivity problems. Autonomous mode is also going to play some role next year. I would suggest two different evaluations for that. One would be a simple description of what their program does. A list could be made of the most common things(Up the ramp, attack human player bins, etc for this year) along with a miscellaneous option with a field to enter a description. The other would be another 1-5 rating of the quality. How fast is it, how effective is it, stuff like that.

I’ll try to help you out in generalities and tell you what worked for me and what didn’t…

When I did WASH in 2001 & 2002 I tried my hardest to keep the fields numerical or time based so the summary sheets could give meaningful information such as average, min, max, etc. I recommend that you do the same. I also found it hard to impose a 1-5 rating for a robot’s performance because unfortunately a robot of quality 4 to me is not a robot of quality 4 to someone else. But I did it anyway and tried to train the scouts. I also found that it helps to assign meanings to the 1-5 ratings, such as 1 for traction means the robot could be pushed sideways, and a 5 means there was no slippage. Stuff like that.

I also found that for match scouting you should keep it as simple & concise as possible. I made the mistake of having too many fields and it just confused the scouts. They didn’t have enough time and they had to rush to finish filling out the forms.

Finally and most importantly, design your system so you can easily and quickly modify the fields you gather. The game is never played out like we first invision it so the scouting system must be able to adapt. It should take no more than 2-3 days to add or remove a field from the DB, web interface, and input programs because that’s all the time you’ll have between regionals to work on it. Trust me, I’ve been there :slight_smile:

Keep up the good work everyone.

Mike

I’ll expand on that just a bit…

Flexibility is the key word you’ve got to run through your head a few times while designing the base system. You have to think about not only how the game may change from Jan->April, but how the game could change from this year -> 2 years from now. If you are planning on keeping any historical data, you will want to be able to either expand/change the game requirements without having to start new databases or totally re-design it every year and then convert tons of old data -> new data.

Just something to think about.

As far as what data to collect it rely depends on next year’s game but there’s standard stuff:

Pitt Scouting:

  • Team #
  • Type of drive (2 whl, 4 whl, tank, crab, …)
  • Drive motors used
  • info about transmission (have/don’t, brake?, how many gears, …)
  • of modules/attachments

  • capabilities (ex. stack)

Match Scouting

  • Team #
  • Partner
  • Opponents
  • Match score
  • QP score
  • Other alliance’s match score
  • Other alliance’s QP score
  • If they won
  • Speed
  • Traction
  • Evaluation of capabilities (ex. how well they stack)
    If they have auto. next year
  • What it does
  • How well it works

With the match and QP scores you could just fill in the match scores and whether or not you won and let the QP score be a calculated field. Or you could just enter Alli he info.

I agree that evaluating capabilities gets very subjective. As said before just using # 1 to 5 doesn’t work because one person’s 5 may be another’s 3. This year my team got around the number thing we had a list of choices N/A, not functioning, marginal, successful, and amazing. This worked pretty good but you couldn’t average or total scores. So I’ve been thinking why not use percentages! They’re universal a 90% in Michigan is the same as a 90% in California. Also, everyone knows what the numbers mean: 60’s - below average/not working well, 70’s - average/works, 80’s above average/works great … With percentages you need to explain the everyone knows what they mean, they’re great!

Just my latest crazy idea
Lindsey

Hey there,
I just registered on the FSN, and there were a couple of things that I think for sure that you should add. For 1, you definately need to know what team each of the members is from. Also, it would help to have a page w/ a list of all of the teams, and possibly the team websites attatched. Probably a good idea to have a link for each team, and to get as much info about them on there as possible. History, past awards, pics, sponsors??? and basic info would probably be good things to include too. If I knew anything about how to work a website, I’d offer to help, but I’m still trying to figure out how to get my picture up on this one!

try to include what their stat would be, this wouldn’t be possible untill comp season though. also if possible give some basic names for their autonomous mode, it’s gonna be back next year. and then put their am into one of those categories.

You need to make this as simple as possilbe. The less number of forms one needs to fill out the better… use some very descriptive fields and leave all reoccuring fields seperate (Ie Robot Name). Hopefully with a system all information will be filled out weeks before a competition and will be available for all teams.

Maybe this post belongs in the other thread, so move it if it more appropriate there.

Make sure you properly seperate the data you’re collecting (and design your database correctly) so that you collect data where & when it makes sense. Data that never changes - team number, team name, etc - should be gathered during the build season. Non-subjective robot data - 2/4wd, speed, stacker/ball collector, etc - should be filled in by teams whenever they feel comfortable revealing that info and by pit scouts. Subjective data - how well the robot collects/scores balls, moves goals, stacks bins, protects stacks, balances the goals, etc - should be collected during matches. Do not trust teams to tell you how well they perform. They either overstate their abilities or flat out lie. Match scoring data should not be collected by the scouts. Either data-mine from FIRST or (if you can’t do it at comps) have a dedicated person do it.

Since this is an area of great interest to me I would like to put in my 2 cents.

First of all, the comments that Mike Soukup made re:scouting are worth rereading - he is making good points from an experience base that few of us have. One key point is that ease-of-recording, simplicity, clarity - whatever you want to call it - is a huge factor in getting high-quality data from multiple sources.

A couple of people have suggested categories of scouting data - I agree, and suggest the following four categories:

  1. ROBOT FEATURES - descriptions of the hardware/software based on observations prior to or during events; can be self-submitted or objectively observed by others
    Example: # of drive motors

  2. ROBOT RATINGS - subjective ratings made based on observations of actual robot performance during practice or matches; require consistent observation and the judgement of dedicated scouts
    Example: Pushing Ability (1=poor, 5=great)

  3. ROBOT RESULTS - specific outcomes that must be observed during practice or matches; require consistent observation by dedicated scouts
    Example: First to the top (yes/no); Defend Top (yes/no); Tallest Stack Made (#bins)

  4. MATCH INFO - basic match data taken from FIRST match listings and match results
    Example: Match Points (#points)

I think all the gathered data will fall into one of these categories. What do you think?

Ken

We actually video-taped all of practice day and all of day one and watched tapes back at the hotel after each event. We didn’t really attempt to make “during the match” scouting, since there is too much going on.

Hint. Hint. FSN might want to get a camera to each regional and do the info dump off-line. This is an unbelievably enormous task. BUt, I suspect that teams would contribute film for the good of the cause.

We believe in quantitative stuff.

Empirical speed of robot is more important that whether the team has a transmission or a particular gear ratio. For last year, we would care about time to top of the ramp during autonomous, time over a short distance (eg 3 meters), time over a long distance (eg 10 meters).

Whatever the game, I think time to the “first objective” is an important stat.
2000: time to far side of field
2001: time to the near side of the bridge
2002: time to the goals in the center
2003: time to the top of the ramp

Rather than having a power stat, you would like to see how teams actually fare against each other in competition. Ie Team XXX pushed Team YYY in match ZZZ. Later, Team YYY pushed Team WWW in match VVV. You can start to form a power class based on who pushed whom.

Robot weight is a really important stat and this should be obtained at the inspection station. I believe FIRST actually records this stat.

I agree with Ken’s Kategories.

On match info, points scored, points allowed, QPs, win/loss.

We also try to accumulate average rank of alliance partners and average rank of opponents (computed at the end of the Qualifying).

Now, if FIRST throws a 4v0 plan against us again, most of the match data would have to be refigured.

Additional Game Specific Match info would be maybe necessary (even though hard to collect):
2000: number of balls robot put into trough
2001: big balls picked up and placed (even if subsequently knocked off), little balls scored in the goal, bridge balanced
2002: balls placed in goal, goals placed in zones
2003: bins stacked (ha. ha.), part of wall knocked over (20%, 50%, 100%), bins descored (10%, …), on ramp

This last would require film.

An additional Kategory: Human Player Performance.