How many times is enough to prove consistency?

[Knufire]

Remember, quantitative data is ALWAYS better than qualitiative. You have a goal in mind (measuring the consistency of robots crossing defenses). How can you measure this with a hard, factual number?

If you have a bunch of qualitative ratings that scouters can choose from, the robot will be at the mercy of the scouter's opinion. Different scouters on your team might have different standards as to what a "good" vs "struggle" crossing is.

I also don't see that many "unsuccessful" crossings happen. An unsuccessful crossing can only happen two ways; you either fail to get over the obstacle and back up, or you get stuck. If you're stuck, you're done for the match unless a partner comes and helps you out, and that probably only counts as one (albeit massive) failure. If you have to back up and try again, you'll probably get the same result (re. definition of insanity) unless you try to Duke of Hazzard it over.

My initial thought was to record not whether the robot crossed or not, but the time it took the robot to cross. Having the average time to cross each obstacle for every team at a tournament would be an extremely valuable piece of data. Downside is that this is probably only practical to record with some sort of electronic scouting system. For this method, I'd also define crossing per the manual definition of a CROSS for clarity.

[Jcarbon]

We generally scout both quantitatively and qualitatively. This year, I think we'll count total crossings and also have experienced scouts judge speed, consistency, and ease of crossing. Then, we can combine the information to make decisions - did they cross a lot but do it hesitantly? Did they do only a few crossings but do each one smoothly and quickly? Of course, whatever system we use will probably change once we see what matches and robots actually look like.

[gblake]

If a team+robot accomplishes something every time they attempt that thing between Wednesday and Sunday, I would say that they were consistently able to accomplish it.

On a slightly more serious note, one big aspect of achieving consistency when doing anything is controlling the independent variables that affect doing that thing.

If you call the thing-to-be-done a process, that process will consume (or be affected by) inputs that are fed into it purposefully (such as battery charge remaining, driver commands, etc.) and by inputs that come from the environment around it (temperature, other robots, debris, friction, etc).

If you want to think about whether other teams' FRC robots can accomplish some process consistently, your thinking needs to include thinking about all the inputs that affect the process, and about how many of those inputs the other team-plus-robot are able to control (how many are possible to control, and how many do they control); and think a little about which of those inputs are the most important.

If the team-plus-robot that you are assessing is doing well at controlling the important variables that can be controlled, then they are likely to perform consistently (predictably) during matches. They (and their allies) will be able to *predict* outcomes as well as can be reasonably expected.

And, I think that being able to make accurate *predictions* is the real topic being discussed here.

With that in mind, seeing another team-plus-robot accomplish some process once could easily convince me that I can rely on their predictions of how well they will perform that process in the future; so long as that one observation was complemented with me knowing that they are controlling the important independent variables that affect the process.

I hope this helps.