Go to Post I would rather Andy Baker be at champs than Will.i.am. - Grim Tuesday [more]
Home
Go Back   Chief Delphi > Competition > Rules/Strategy
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
View Poll Results: How many times are necessary to determine consistency?
Defenses: 1-3 1 1.35%
Defenses: 4-6 14 18.92%
Defenses: 7-10 18 24.32%
Defenses: 11-20 18 24.32%
Defenses: All 27 36.49%
Climbing: 1-3 4 5.41%
Climbing: 4-8 27 36.49%
Climbing: All 33 44.59%
Multiple Choice Poll. Voters: 74. You may not vote on this poll

Closed Thread
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 26-01-2016, 21:43
tindleroot tindleroot is offline
Free Scouting Help to All
AKA: Casey LeeVan
FRC #4272 (Maverick Boiler Robotics)
Team Role: Mentor
 
Join Date: Mar 2014
Rookie Year: 2007
Location: Indiana
Posts: 593
tindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond repute
How many times is enough to prove consistency?

I just realized that you could keep all scouting data about defense crossing and climbing in one chart for all teams. In doing so, you could have each cell representing how a team handles a particular defense. For example, every match, you can rate a crossing as "Good", "Struggle", or "Unsuccessful", perhaps with different colors to represent each, or different locations within the cell. This would really help those with paper-based systems as one scout could quickly record all of the information (after the match, based on other scouts' findings) into one chart that visually represents robots' abilities and success rates. I'll post an example when I get time to do it.

How many times do you think are required, for defenses and for climbing, for a robot to prove that they are capable of consistently passing a defense? (i.e. how much information would be necessary on a chart?)
__________________
2012-2013: FTC 4601 (student)
2014-2015: FRC 135 (student) - 5 banners in 2 years!
2016-present: FRC 4272 (mentor)

Thanks to all of our alliance partners!
11, 51, 68, 201, 234, 868, 1108, 1241, 1760, 3147, 3301, 3487, 3865, 4269, 4485, 4580, 5188

  #2   Spotlight this post!  
Unread 26-01-2016, 22:17
Turing'sEgo Turing'sEgo is offline
Registered User
no team
 
Join Date: Jan 2016
Rookie Year: 2010
Location: Boulder
Posts: 47
Turing'sEgo can only hope to improve
Re: How many times is enough to prove consistency?

Well you could decompose this into a pure statistics problem with hypothesis testing and what not. I remember exactly none of that, sadly. That would be one thorough scouting analysis. Not sure you'd be able to get enough data throughout the competition, however.
  #3   Spotlight this post!  
Unread 26-01-2016, 22:24
prozack19's Avatar
prozack19 prozack19 is offline
Registered User
AKA: Zackary Peery
FRC #0056 (R.O.B.B.E.)
Team Role: Mentor
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Middlesex
Posts: 36
prozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant future
Send a message via MSN to prozack19
Thumbs up Re: How many times is enough to prove consistency?

Quote:
Originally Posted by tindleroot View Post
I just realized that you could keep all scouting data about defense crossing and climbing in one chart for all teams. In doing so, you could have each cell representing how a team handles a particular defense. For example, every match, you can rate a crossing as "Good", "Struggle", or "Unsuccessful", perhaps with different colors to represent each, or different locations within the cell. This would really help those with paper-based systems as one scout could quickly record all of the information (after the match, based on other scouts' findings) into one chart that visually represents robots' abilities and success rates. I'll post an example when I get time to do it.
Im actually working on one for my team. everything has a number representation and will all be graphed and the information will all then be put onto the same sheet all in excel.
__________________
FRC 869
2013-web design
2014- lead scout/strategist
2015- lead scout/strategist, hp, chairmans presenter
2016- lead scout/strategist
FRC 56 2016- present
scouting and strategy mentor
  #4   Spotlight this post!  
Unread 26-01-2016, 22:25
prozack19's Avatar
prozack19 prozack19 is offline
Registered User
AKA: Zackary Peery
FRC #0056 (R.O.B.B.E.)
Team Role: Mentor
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Middlesex
Posts: 36
prozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant futureprozack19 has a brilliant future
Send a message via MSN to prozack19
Re: How many times is enough to prove consistency?

IMO consistency needs do be done 90% or more to show that they are actually capable of doing it.
__________________
FRC 869
2013-web design
2014- lead scout/strategist
2015- lead scout/strategist, hp, chairmans presenter
2016- lead scout/strategist
FRC 56 2016- present
scouting and strategy mentor
  #5   Spotlight this post!  
Unread 26-01-2016, 22:49
evanperryg's Avatar
evanperryg evanperryg is offline
IT'S THE BUMP N' DUMP
AKA: Evan Grove
FRC #4536 (The Minutebots)
Team Role: Mentor
 
Join Date: Apr 2013
Rookie Year: 2011
Location: Minneapolis, MN
Posts: 656
evanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond reputeevanperryg has a reputation beyond repute
Re: How many times is enough to prove consistency?

Quote:
Originally Posted by tindleroot View Post
How many times do you think are required, for defenses and for climbing, for a robot to prove that they are capable of consistently passing a defense? (i.e. how much information would be necessary on a chart?)
The inherent problem with counting crosses is that teams will cross the easy ones a lot more than, say, the portcullis. I think a more appropriate way of testing would be a count of crosses, next to a count of times a team got stuck, perhaps a +/- score, consisting of (successful crosses - (k)(failed cross), with a k value adjusted based on how agressively you want to eliminate teams for failures. IMO a team that can't cross the group B and/or D defenses 100% of the time isn't pickable, regardless of what else they do. With no more than 1 (that one being the low bar, hypothetically) reliable method to traverse the outer works, you have a huge bottlenecking problem, especially considering how congested that area of the field is (hypothetically) going to be.
__________________
FRCDesigns Contributor | "There is only one corner of the universe you can be certain of improving, and that's your own self." -Aldous Huxley
2012-2016 | FRC Team 2338: Gear it Forward
2013
Wisconsin Regional Winner 2014 Midwest Regional Finalist 2015 Midwest Regional Chairman's Award, Finalist, Archimedes Division Champion, IRI Semifinalist 2016 Midwest Regional Chairman's Award, Finalist, Archimedes Division Gracious Professionalism Award, R2OC Winner
2015 | FTC Team 10266: Mach Speed
2015
Highland Park Qualifier Winner, Motivate Award
2017-???? | FRC Team 4536: The Minutebots

Thanks to the alliances and friends I've made along the way: 33 74 107 111 167 171 234 548 1023 1089 1323 1625 1675 1732 1756 2064 2077 2122 2202 2358 2451 2512 2826 3936 3996 4039 4085 4241 5006 5401 5568 5847 5934
  #6   Spotlight this post!  
Unread 26-01-2016, 23:43
z_beeblebrox's Avatar
z_beeblebrox z_beeblebrox is offline
Custom User Title
AKA: Cal
FRC #4183 (Bit Buckets)
Team Role: Alumni
 
Join Date: Jan 2012
Rookie Year: 2012
Location: Cambridge MA
Posts: 811
z_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond reputez_beeblebrox has a reputation beyond repute
Re: How many times is enough to prove consistency?

Another important consideration is when failures occur, since teams often improve over the event. I'd feel better picking a team with 5 failures in their first couple of matches and none later in the event than a team with 3 failures in matches 3, 9 and 10.
__________________
2012 Utah Regional Rookie All-Star
2013 Phoenix Regional Judge's Award for "design process and prototyping"
2014 Hub City Regional Quality Award, Arizona Regional Excellence in Engineering Award
2015 Arizona East Regional Creativity Award, Winner
2016 Arizona North Regional Finalist, Arizona West Excellence in Engineering Award, Finalist
  #7   Spotlight this post!  
Unread 27-01-2016, 00:07
Chief Hedgehog's Avatar
Chief Hedgehog Chief Hedgehog is offline
Mentor
FRC #4607 (C.I.S.)
Team Role: Coach
 
Join Date: May 2013
Rookie Year: 2012
Location: Becker, Minnesota
Posts: 538
Chief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond repute
Re: How many times is enough to prove consistency?

Once is luck; three times is a trend. However, to accomplish a task without human error is consistency.

Since this is a game involving humans, there is no true measure of consistency.
__________________

"An error does not become a mistake until you refuse to correct it" ~JFK
  #8   Spotlight this post!  
Unread 27-01-2016, 00:18
asid61's Avatar
asid61 asid61 is offline
Registered User
AKA: Anand Rajamani
FRC #0115 (MVRT)
Team Role: Mechanical
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Cupertino, CA
Posts: 2,215
asid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond reputeasid61 has a reputation beyond repute
Re: How many times is enough to prove consistency?

If they try and fail, that shows inconsistency. If they do not try at all some matches, but succeed in others, that shows consistency.
I think a scout can look for how "naturally" they vault the defenses to determine consistency with only a few trials rather than depend on pure numbers (although those are important as well).
__________________
<Now accepting CAD requests and commissions>

  #9   Spotlight this post!  
Unread 27-01-2016, 01:23
tindleroot tindleroot is offline
Free Scouting Help to All
AKA: Casey LeeVan
FRC #4272 (Maverick Boiler Robotics)
Team Role: Mentor
 
Join Date: Mar 2014
Rookie Year: 2007
Location: Indiana
Posts: 593
tindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond reputetindleroot has a reputation beyond repute
Re: How many times is enough to prove consistency?

Quote:
Originally Posted by Chief Hedgehog View Post
Once is luck; three times is a trend. However, to accomplish a task without human error is consistency.

Since this is a game involving humans, there is no true measure of consistency.
While you present a great theory, scouting depends on some sort of measurement. I am trying to consider a good compromise in order to minimize data accumulated and still determine consistency to a predictable degree. For me, if I saw a robot cross 5 times successfully, I would trust them to continue doing it (as long as those 5 crosses were not among a sea of failures, and the trend does not change sporadically throughout the competition). Hanging, I would want to make sure they make it every time.

Honestly, I don't expect teams to be particularly inconsistent with the defenses they are intending to cross. Either they will cross or they won't. But now that I say that, the opposite will probably happen.
__________________
2012-2013: FTC 4601 (student)
2014-2015: FRC 135 (student) - 5 banners in 2 years!
2016-present: FRC 4272 (mentor)

Thanks to all of our alliance partners!
11, 51, 68, 201, 234, 868, 1108, 1241, 1760, 3147, 3301, 3487, 3865, 4269, 4485, 4580, 5188

  #10   Spotlight this post!  
Unread 27-01-2016, 01:25
pandamonium's Avatar
pandamonium pandamonium is offline
Registered User
FRC #1836 (Milken Knights)
Team Role: Mentor
 
Join Date: Jan 2006
Rookie Year: 2005
Location: Los Angeles, CA
Posts: 407
pandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond reputepandamonium has a reputation beyond repute
Re: How many times is enough to prove consistency?

https://www.youtube.com/watch?v=yg_ssvGBqEs
  #11   Spotlight this post!  
Unread 27-01-2016, 02:47
SoftwareBug2.0's Avatar
SoftwareBug2.0 SoftwareBug2.0 is offline
Registered User
AKA: Eric
FRC #1425 (Error Code Xero)
Team Role: Mentor
 
Join Date: Aug 2004
Rookie Year: 2004
Location: Tigard, Oregon
Posts: 485
SoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant futureSoftwareBug2.0 has a brilliant future
Re: How many times is enough to prove consistency?

Quote:
Originally Posted by evanperryg View Post
The inherent problem with counting crosses is that teams will cross the easy ones a lot more than, say, the portcullis. I think a more appropriate way of testing would be a count of crosses, next to a count of times a team got stuck, perhaps a +/- score, consisting of (successful crosses - (k)(failed cross), with a k value adjusted based on how agressively you want to eliminate teams for failures.
If only someone would develop an area of statistics that was devoted to degrees of belief. We could even name it after my favorite 18th century Presbyterian minister.
  #12   Spotlight this post!  
Unread 27-01-2016, 03:04
Chief Hedgehog's Avatar
Chief Hedgehog Chief Hedgehog is offline
Mentor
FRC #4607 (C.I.S.)
Team Role: Coach
 
Join Date: May 2013
Rookie Year: 2012
Location: Becker, Minnesota
Posts: 538
Chief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond reputeChief Hedgehog has a reputation beyond repute
Re: How many times is enough to prove consistency?

Quote:
Originally Posted by tindleroot View Post
While you present a great theory, scouting depends on some sort of measurement. I am trying to consider a good compromise in order to minimize data accumulated and still determine consistency to a predictable degree. For me, if I saw a robot cross 5 times successfully, I would trust them to continue doing it (as long as those 5 crosses were not among a sea of failures, and the trend does not change sporadically throughout the competition). Hanging, I would want to make sure they make it every time.

Honestly, I don't expect teams to be particularly inconsistent with the defenses they are intending to cross. Either they will cross or they won't. But now that I say that, the opposite will probably happen.
Five times is great. But most events have 7-10 rounds. Did your team accomplish this task 5 times out of 7, or 10? Did they do this within traffic, or did this do this with impunity?

Consistency is great - but your scouting better be ready to address the other issues involved or the better scouting from another alliance may beat you.
__________________

"An error does not become a mistake until you refuse to correct it" ~JFK

Last edited by Chief Hedgehog : 27-01-2016 at 03:06.
  #13   Spotlight this post!  
Unread 27-01-2016, 03:05
Knufire Knufire is offline
Rose-Hulman Institute of Technology
no team
Team Role: College Student
 
Join Date: Sep 2009
Rookie Year: 2010
Location: Terre Haute, IN
Posts: 736
Knufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond reputeKnufire has a reputation beyond repute
Re: How many times is enough to prove consistency?

Remember, quantitative data is ALWAYS better than qualitiative. You have a goal in mind (measuring the consistency of robots crossing defenses). How can you measure this with a hard, factual number?

If you have a bunch of qualitative ratings that scouters can choose from, the robot will be at the mercy of the scouter's opinion. Different scouters on your team might have different standards as to what a "good" vs "struggle" crossing is.

I also don't see that many "unsuccessful" crossings happen. An unsuccessful crossing can only happen two ways; you either fail to get over the obstacle and back up, or you get stuck. If you're stuck, you're done for the match unless a partner comes and helps you out, and that probably only counts as one (albeit massive) failure. If you have to back up and try again, you'll probably get the same result (re. definition of insanity) unless you try to Duke of Hazzard it over.

My initial thought was to record not whether the robot crossed or not, but the time it took the robot to cross. Having the average time to cross each obstacle for every team at a tournament would be an extremely valuable piece of data. Downside is that this is probably only practical to record with some sort of electronic scouting system. For this method, I'd also define crossing per the manual definition of a CROSS for clarity.
__________________
Team 469: 2010 - 2013
Team 5188: 2014 - 2016
NAR (VEX U): 2014 - Present
  #14   Spotlight this post!  
Unread 27-01-2016, 10:22
Jcarbon Jcarbon is offline
Registered User
FRC #3322 (Eagle Imperium)
Team Role: Leadership
 
Join Date: Jan 2016
Rookie Year: 2014
Location: Ann Arbor, Mi
Posts: 62
Jcarbon has a spectacular aura aboutJcarbon has a spectacular aura about
Re: How many times is enough to prove consistency?

We generally scout both quantitatively and qualitatively. This year, I think we'll count total crossings and also have experienced scouts judge speed, consistency, and ease of crossing. Then, we can combine the information to make decisions - did they cross a lot but do it hesitantly? Did they do only a few crossings but do each one smoothly and quickly? Of course, whatever system we use will probably change once we see what matches and robots actually look like.
  #15   Spotlight this post!  
Unread 27-01-2016, 14:19
gblake's Avatar
gblake gblake is offline
6th Gear Developer; Mentor
AKA: Blake Ross
no team (6th Gear)
Team Role: Mentor
 
Join Date: May 2006
Rookie Year: 2006
Location: Virginia
Posts: 1,934
gblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond reputegblake has a reputation beyond repute
Re: How many times is enough to prove consistency?

If a team+robot accomplishes something every time they attempt that thing between Wednesday and Sunday, I would say that they were consistently able to accomplish it.

On a slightly more serious note, one big aspect of achieving consistency when doing anything is controlling the independent variables that affect doing that thing.

If you call the thing-to-be-done a process, that process will consume (or be affected by) inputs that are fed into it purposefully (such as battery charge remaining, driver commands, etc.) and by inputs that come from the environment around it (temperature, other robots, debris, friction, etc).

If you want to think about whether other teams' FRC robots can accomplish some process consistently, your thinking needs to include thinking about all the inputs that affect the process, and about how many of those inputs the other team-plus-robot are able to control (how many are possible to control, and how many do they control); and think a little about which of those inputs are the most important.

If the team-plus-robot that you are assessing is doing well at controlling the important variables that can be controlled, then they are likely to perform consistently (predictably) during matches. They (and their allies) will be able to *predict* outcomes as well as can be reasonably expected.

And, I think that being able to make accurate *predictions* is the real topic being discussed here.

With that in mind, seeing another team-plus-robot accomplish some process once could easily convince me that I can rely on their predictions of how well they will perform that process in the future; so long as that one observation was complemented with me knowing that they are controlling the important independent variables that affect the process.

I hope this helps.
__________________
Blake Ross, For emailing me, in the verizon.net domain, I am blake
VRC Team Mentor, FTC volunteer, 5th Gear Developer, Husband, Father, Triangle Fraternity Alumnus (ky 76), U Ky BSEE, Tau Beta Pi, Eta Kappa Nu, Kentucky Colonel
Words/phrases I avoid: basis, mitigate, leveraging, transitioning, impact (instead of affect/effect), facilitate, programmatic, problematic, issue (instead of problem), latency (instead of delay), dependency (instead of prerequisite), connectivity, usage & utilize (instead of use), downed, functionality, functional, power on, descore, alumni (instead of alumnus/alumna), the enterprise, methodology, nomenclature, form factor (instead of size or shape), competency, modality, provided(with), provision(ing), irregardless/irrespective, signage, colorized, pulsating, ideate

Last edited by gblake : 27-01-2016 at 14:29.
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 11:28.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi