I just realized that you could keep all scouting data about defense crossing and climbing in one chart for all teams. In doing so, you could have each cell representing how a team handles a particular defense. For example, every match, you can rate a crossing as “Good”, “Struggle”, or “Unsuccessful”, perhaps with different colors to represent each, or different locations within the cell. This would really help those with paper-based systems as one scout could quickly record all of the information (after the match, based on other scouts’ findings) into one chart that visually represents robots’ abilities and success rates. I’ll post an example when I get time to do it.
How many times do you think are required, for defenses and for climbing, for a robot to prove that they are capable of consistently passing a defense? (i.e. how much information would be necessary on a chart?)
Well you could decompose this into a pure statistics problem with hypothesis testing and what not. I remember exactly none of that, sadly. That would be one thorough scouting analysis. Not sure you’d be able to get enough data throughout the competition, however.
Im actually working on one for my team. everything has a number representation and will all be graphed and the information will all then be put onto the same sheet all in excel.
The inherent problem with counting crosses is that teams will cross the easy ones a lot more than, say, the portcullis. I think a more appropriate way of testing would be a count of crosses, next to a count of times a team got stuck, perhaps a +/- score, consisting of (successful crosses - (k)(failed cross), with a k value adjusted based on how agressively you want to eliminate teams for failures. IMO a team that can’t cross the group B and/or D defenses 100% of the time isn’t pickable, regardless of what else they do. With no more than 1 (that one being the low bar, hypothetically) reliable method to traverse the outer works, you have a huge bottlenecking problem, especially considering how congested that area of the field is (hypothetically) going to be.
Another important consideration is when failures occur, since teams often improve over the event. I’d feel better picking a team with 5 failures in their first couple of matches and none later in the event than a team with 3 failures in matches 3, 9 and 10.
If they try and fail, that shows inconsistency. If they do not try at all some matches, but succeed in others, that shows consistency.
I think a scout can look for how “naturally” they vault the defenses to determine consistency with only a few trials rather than depend on pure numbers (although those are important as well).
While you present a great theory, scouting depends on some sort of measurement. I am trying to consider a good compromise in order to minimize data accumulated and still determine consistency to a predictable degree. For me, if I saw a robot cross 5 times successfully, I would trust them to continue doing it (as long as those 5 crosses were not among a sea of failures, and the trend does not change sporadically throughout the competition). Hanging, I would want to make sure they make it every time.
Honestly, I don’t expect teams to be particularly inconsistent with the defenses they are intending to cross. Either they will cross or they won’t. But now that I say that, the opposite will probably happen.
If only someone would develop an area of statistics that was devoted to degrees of belief. We could even name it after my favorite 18th century Presbyterian minister.
Five times is great. But most events have 7-10 rounds. Did your team accomplish this task 5 times out of 7, or 10? Did they do this within traffic, or did this do this with impunity?
Consistency is great - but your scouting better be ready to address the other issues involved or the better scouting from another alliance may beat you.
Remember, quantitative data is ALWAYS better than qualitiative. You have a goal in mind (measuring the consistency of robots crossing defenses). How can you measure this with a hard, factual number?
If you have a bunch of qualitative ratings that scouters can choose from, the robot will be at the mercy of the scouter’s opinion. Different scouters on your team might have different standards as to what a “good” vs “struggle” crossing is.
I also don’t see that many “unsuccessful” crossings happen. An unsuccessful crossing can only happen two ways; you either fail to get over the obstacle and back up, or you get stuck. If you’re stuck, you’re done for the match unless a partner comes and helps you out, and that probably only counts as one (albeit massive) failure. If you have to back up and try again, you’ll probably get the same result (re. definition of insanity) unless you try to Duke of Hazzard it over.
My initial thought was to record not whether the robot crossed or not, but the time it took the robot to cross. Having the average time to cross each obstacle for every team at a tournament would be an extremely valuable piece of data. Downside is that this is probably only practical to record with some sort of electronic scouting system. For this method, I’d also define crossing per the manual definition of a CROSS for clarity.
We generally scout both quantitatively and qualitatively. This year, I think we’ll count total crossings and also have experienced scouts judge speed, consistency, and ease of crossing. Then, we can combine the information to make decisions - did they cross a lot but do it hesitantly? Did they do only a few crossings but do each one smoothly and quickly? Of course, whatever system we use will probably change once we see what matches and robots actually look like.
If a team+robot accomplishes something every time they attempt that thing between Wednesday and Sunday, I would say that they were consistently able to accomplish it.
On a slightly more serious note, one big aspect of achieving consistency when doing anything is controlling the independent variables that affect doing that thing.
If you call the thing-to-be-done a process, that process will consume (or be affected by) inputs that are fed into it purposefully (such as battery charge remaining, driver commands, etc.) and by inputs that come from the environment around it (temperature, other robots, debris, friction, etc).
If you want to think about whether other teams’ FRC robots can accomplish some process consistently, your thinking needs to include thinking about all the inputs that affect the process, and about how many of those inputs the other team-plus-robot are able to control (how many are possible to control, and how many do they control); and think a little about which of those inputs are the most important.
If the team-plus-robot that you are assessing is doing well at controlling the important variables that can be controlled, then they are likely to perform consistently (predictably) during matches. They (and their allies) will be able to predict outcomes as well as can be reasonably expected.
And, I think that being able to make accurate predictions is the real topic being discussed here.
With that in mind, seeing another team-plus-robot accomplish some process once could easily convince me that I can rely on their predictions of how well they will perform that process in the future; so long as that one observation was complemented with me knowing that they are controlling the important independent variables that affect the process.