Chief Delphi - [FIRST EMAIL] Stop Build Day Survey

Quote:

Originally Posted by Karthik (Post 1605727)

I believe Issac Rife tried to collect this data a few years back.

Spending my off Friday writing a long note about a long topic.... Not a ton of content I haven't posted elsewhere, but this 23 page thread really felt like it needed another wall of text, so here goes...

In 2012, I was a mentor with 33 (2005-2013 seasons) and Jim asked myself and another mentor to do some survey work to understand the importance of a practice robot and practice field towards exceling in FRC.

Some background:
33 has generally been a very good team with typical performances pre 2011 usually making it to the semi-finals or occasionally the finals, but they did not win a regular season event from 2005 through 2010 (to my knowledge). During that time, Jim and the other mentors were aware that the best teams were building practice robots, but we couldn't quite get one made. In 2007, we built a really close one probably about 75% accurate with the biggest difference being it did not have the ramps. The competition team got about 10 hours of practice (actual operating hours) before the robot self destructed. That did very well, and we came crazy close to beating the eventual World champions that year. 2008, no practice robot, and the driver (2nd year) nearly destroyed our season in the first match as he hit the overpass too fast. That was a very good robot, but not the contender we were looking for. 2009, very complicated robot. Pretty tricky to drive. Our driver never really got the hang of it. 2010, Made a practice robot that was pretty close say 80-90%. It was missing its hanger, and did not go over the bumps as well, but the driver did get practice gathering and scoring. That robot was a real contender. 2011, renewed focus on making an even better practice robot. That practice bot was very close on the order of 95% similar. The team was practicing on a 2/3 scale field and got a lot of practice. This was the first year the team started winning competitions. We had an issue with the minibot firing device that was only on the comp bot, and was incredibly frustrating to diagnose. 2012- An even hirer focus on making comp bot and practice bot "the same" was put into effect. Practice bot was working great, but at first competition, comp bot had a shooting issue that did not get resolved until the last round of qualifying. This was the year Jim asked me to conduct the survey as we were still practicing on a 2/3 practice field.

2012 Survey Portion:
That survey asked teams if they had a practice robot, estimate of how similar, did they have a practice field or access to one, number of mentors, number of "core mentors", number of students, and number of students engaged in build.
The survey was focused on the top 25 OPR teams in each division. We were able to talk to 85 of those 100 teams. 71/85 teams had some sort of practice robot. The most basic was for a team to use an old robot for driving practice. This was done by a couple teams that use very similar chassis year to year and often have very good FRC success.

The general trend from that was the better the team, the closer the practice bot was to the competition robot. There was a mix of building 2 copies of the robot, building practice first, and building practice second. Teams with a lot of automated/CNC/waterjet tended to build two copies and try to keep them the same. 1114 was a neat team to talk to as they tended to build their practice first, and learn, and then very quickly build their competition robot towards the later part of build season. I believe this is still true, though I see more and more CNC/Waterjet on their bots each year which makes me wonder how that has influenced this practice. For instance, Simbot SS from 2008 appeared to be fully rough fabricated vs. automated machining.
********************************
Additional thoughts:
Jim's paper's thoughts and Causation vs. Correlation
I have not conducted such a survey since then, but I believe Jim has conducted other surveys which we have discussed results and I know are background to a lot of the hypothesis that are in his paper. IE, he is aware correlation is not necessarily causation, but the other anecdotes and data over the years tends to support those ideas. Jim is a coach which means every competition he interfaces with roughly 20 to 24 other coaches during qualifying. Multiply that by 2 districts and 1 MSC and 1 World championship (total of about 75 to 100 coaches) per year times 8 years since districts has started, and you have about 500 to 800 coach to coach interactions. In line, he talks with them about the match, but also about how their season is going, and what kind of challenges they are having. Sure, a lot of those repeats, but when talking to the same coach and seeing them having a good year vs. last year's bad year, you can get a much better idea of what they did to improve (or in some cases what they did that is causing them trouble).
Most mentors that are not coaches do not get this level of interface and diversity of team interface. We spend a lot of time with our team. Some time talking to friends, and probably some time talking with teams we admire. As a coach, you are forced to talk with that diversity.
***********************************************

What I do now and My thoughts on SBD:

I left 33 after 2013 season as my job had moved sites, and my job role had changed to where I could not be the kind of mentor for the team that I wanted to be. I also was taking on increasing responsibility within FiM with regards to LRI and LRI prep. I have been helping 469 the last couple years, but frankly I am a crappy mentor as my priorities are now family, Job, FiM events, then 469 students, then 469 team. If you ask my wife or use hours to equate to priorities, then it would be Job, FiM, Family, 469 students, 469 team January through end of April then Job, Family, FIRST the rest of the year.

My current thoughts on SBD are as and FRC Enthusiast, LRI/Volunteer, then mentor:
As an enthusiast, I would say get rid of SBD. I think the robots would get better overall, the competition would get better, especially in early season, and I think you could get something more television worthy which would do wonders for Spreading knowledge about FRC.

As an LRI, in the district system. I think I get to interface with a lot more teams then most mentors. This is especially true of teams that are struggling to pass inspection. The "second event" portion of the season (weeks 4 and beyond) tend to go much smoother than the first couple of weeks. There are few things more rewarding than seeing a team make the improvements to get some "success" during an event or competition. This is not necessarily winning the event, but it usually involves getting something to work that has not worked before.

I personally believe, having talked with the teams, that they would do better with more access (IE weekly chunk of time), but some of them might actually do worse if the SBD was removed al together (though I am a general proponent of removing it). The reason for this is the SBD helps teams with an intermediate deadline that is not the competition itself. Having participated in Solarcar, Supermileage, and volunteered at SAE competitions, I can tell you that a decent chunk of teams procrastinate and are still trying to get the car running at competition. This is because their view is that the design will work as long as they have it working by competition. Because of this, they end up with Student Syndrome which can turn into all nighters the week of the competition, only to be exhausted, cranky, and frustrated at the event. I believe this would happen with FRC if SBD was removed, and might exacerbate the condition which can be dangerous when teams are driving 4 hours to get to the closest competition. There was a deadly incident of such in FSAE. I believe similar such crashes have occurred in Solarcar whose event is typically driving cross country for long periods of time in slow moving (read boring) convoys. While that was the only 1 incident in FSAE history that I know of, I can't help but think that there have been many other close calls over the years. With FRC having an order of magnitude more teams, and a couple orders of magnitude more competitions, I wonder what the additional dice rolls might result in.
While I personally would like a fully open season, IE stop the Stop build... I would have a hard time recommending it knowing that it would bring harm (possibly lethal harm) to some in the community.

Developing Developers:

I am 110% behind adding additional access though as I believe one of the biggest disservice that we do to many FRC students is get rid of the "development" period of an FRC project. I worked in Auto at an OEM for 8 years, and most of that engineering is development engineering. IE the Advanced Vehicle engineers already have layed out the concept and configuration. As a Design and Release engineer, you are to develop that design into a refined product ready to unleash on the masses. In my current job, planning in periods for development is often met with "oh you want to do a Science experiement" sarcasm as there is a fear of long development periods. Management often takes excite in pulling out the "Sometimes you have to shoot the engineer" catch phrase. While I do believe some may tinker too long, I really think development and fixing and improving issues is truly what makes great products.

If you have watched slingshot, imagine if Dean and the team at DEKA were only allowed to design the first prototype, and then it would either go into production, or not....
Imagine if Rocket engines were designed, and had to work on the first shot as opposed to all the trial runs in the test cell.
Imagine that Rolls Royce put passengers on the first flight of a newly released engine...
Have you ever beta tested a game or piece of software? Ever played an AP that was put out in the market too soon.

Why do we expect students to get it right the first shot? Why do we get upset with those that give their students a taste of development.

Cheating our students of the opportunity to make or design something, fail, then figure out what it takes to make it better is really cheating them of a greater experience.

The great teams are giving that experience. As I said earlier, I have not conducted another survey on practice robots, but instead, I try to visit a team for a couple hours and see how they operate. I have usually targeted the better teams in FRC to understand what their "magic" is. Some are driven by an extremely talented, driven, passionate mentor that will not stop until the teams achieves excellence. Most of a the really good teams have a core group of mentors (3+) that have at least a couple of those attributes. The usual mode for most teams that keep it to a healthy competitiveness (as judged by me) is that a mentor starts an effort and does well with it to the point that it inspires some students to get more engaged and model that behavior. The mentor then works with the student to build up that skill set if they are around for more years until the student is nearly as skilled as the mentor by the time they graduate. Teams that continue to do well year after year will then have students take on the bulk of that effort as the team grows, and the mentors move into a role of advisor/decision maker in that area. Within a couple years of continuous dedicated improvement, you have a juggernaut of a team with the students doing the bulk of the work alongside mentors continuously improving an area or branching into a new area the team has been relatively weak at. This is why many mentors that have developed their students to know how to iterate and develop a design successfully get upset when they are accused of doing the work for the student. When you have invested a couple years in building the confidence and skill set (sometimes skill set then confidence) of a student to see it damaged by a callous statement, it will make your blood boil...
*******************************************
Practice... yes practice

In Outliers, Gladwell often entions 10,000 hrs. of deliberate practice turning into expertise. Not 10,000 hrs of doing the same thing the same way, but 10,000 hrs. focused on getting better at something. I think that with each order of magnitude of deliberate practice/experience, we get better noticeably better than those with an order of magnitude less.

You can do this exercise with any skill you do not have a natural talent for previous experience. For instance, throwing darts or throwing cards in a hat.
For darts, pick a distance where you can hit the board and aim for a specific spot (say bullseye). Your first 10 darts are likely to be all over the place. If you continue to throw until you have thrown 100 darts, I wm willing to bet that you last 10 darts have a much closing grouping than your first 10. If you continued over a week throwing 100 darts a night for 10 nights, by the time you get to 1,000 darts, you likely are getting way way tighter grouping and probably hitting most shots close to the bullseye. Of note, It only take about 1 minute to throw 9 darts in 3 dart groups treading back and fourth, so 100 darts is really only about a 10-15 minute exercise. 1000 darts would be just under 2 hours of your life. By this point, you are still very much a novice at throwing darts, but you will be much better than you were at 10 darts. If you continued that trend over a year say throwing on weeknights 100 darts every two weeks you would hit a new thousand, or you would have thrown roughly 10,000 darts 2 months later. By this point, you would likely be considerably better than that 1,000 dart thrower (but again, you are only about 20 hours into it). I pause at this point, because you will have illustrated what what many of the top 5% of FRC teams get for practice/exerience going into the Championship. They will have competed in about 60 real matches or 2 hours of experience, and likely double that for practice matches and test and tune sessions with their robot and maybe some focused practice. This equates to about 4 hours of actual driving time, and 10 to 20 hours of dealing with getting the robot ready for matches or repairing your practice bot due to thrown chains or xyz. If the total stick times equates to about 120 real matches, you are now a 10X more experience than that team that went to one regional, and got their robot working for 8 qualifying matches and was picked up by a team that was knocked out in the QFs....
*The top 1% (IE FRC top 25 caliber) will roughly double that amount of stick time with the majority of it spent on practice including running specific drills and scenarios to see if they can build that one extra stack, or hang that 9th tube which completes another row or shave off 6 seconds for the line-up of that end game. Mind you we are only talking now 8 hours of stick time and likely 40 hours of concentrated practice. (as a reminder, 8 hours per week x 6 weeks = 48 which is greater than 40...).

*That top 0.1% or top 3-5 teams each year likely see 2-5X that. Or 16 to 40 hours of stick time. Still within that 48 hours of acess we are talking about, though very unlikely to occur with a robot as you have now likely worn out several thing and had to fix/repair them. This 16 to 40 hours of stick-time likely equates to 64 to 120 hours dedicated towards practice. They are definitely mixing up specific skill based drills and 2 minute periods.

I bring this up because I have seen numerous times where a team struggling to hang 2 tubes when they start can get 2 rows within a couple hours.
I have seen a team struggling to build a stack get to where they are consistently at 2 stacks within a couple hours.
I have watched a team struggling to get 50% and 2 cycles of Frisbees hit 75% with 4+ cycles after a couple hours.

FYI: To get very god at Darts, it is recommended to throw 500/weeknight (roughly 1 hour) every night for a year 125,000 darts and 200 hrs. If you still like playing darts, and want o become world champion, up it to 1,000 per night and another 10 hours on the weekend or 500,000 per year and roughly 1,000 hrs/year. Acording to the 10,000 theory, you would be an expert in 10 years. Though realistically, you would appear to be an expert to many after that first year. While this seems crazy, there are a lot of people that put in that much time in video games, pool table, darts, or other sports while taking a full course load in college. At the end of 4 years, they have literally devoted 4,000+ hours to those efforts.

********************************************

So how does this relate to SBD?

I think you should keep it, but add the access windows all season long. My recommendation is 6 hours, but I am fine with Jim's 8 will be enough that teams will get an additional 24 to 64 hours of robot access during the season depending how far they make it.

Before their first event, they will have put the robot in a bag, and should have a discussion to plan out what their new priorities are before the first event. Is it get it moving? Is it giving the programmers 4 hours to calibrate all the sensors and have a basic auton? Is it to tweak the gripper that just isn't gripping thing? Is it a chance to give the driver some needed practice?

Before their second event (it is trending some the majority of teams actually have a second event), they will then have intimate knowledge of what worked and didn't work on their robot. They will get the opportunity to make that change/fix they desperately wanted at the last competition.

Should they make it to a 3rd event, they will have hopefully arranged some practice with some other teams.

The window won't stop the great teams from building extra robots. As many have stated, they would utilize them as programming assets and would not want to wear out their robot (even if you got rid of weight, with a 120 AMP main breaker and single battery, you really don't want a 200 lb gorilla out of the floor unless pushing/holding position is mandatory towards your strategy).

The windows will let more teams get that development experience that is critical towards making and improving their students and their team. It should also let the programmers and drivers get some much needed access so that Auton is less boring, and drivers are less crappy (yes most drive teams suck when they start out. Some continue to suck.) Keeping the SBD let's the marketers keep their catch phrase, and will help some from possibly hurting themselves.

*The practice schedules I reference are estimates from talking with teams about how often and what kind of practice they do. These discussions often very from the fairly reasonable we practice 3 nights a week for about 1 hour and 2-3 hours on Saturdays though we are in the shop 3 hrs. per night 4 nights a week and all day Saturday... To "We practice hundreds of hours". I have had students give me figures that are absolutely laughable. IE I spent 600+ hours during build season in the shop which would be over 13 hours on average. Over six weeks, If they spent 20 hours Saturday, and 20 hours Sunday, they would still need to spend 12 hours each weeknight in the shop to hit their 600 hours... I have had a 20 hour day in the shop and cannot imagine what 12 of those in a build season would do to me.