paper: Stop the Stop Build

[techtiger1]

#Zondag4President

I really like the points people are making about withholding rules and B & T with the 8 hr time limit. Seems like Dr. Joe might have a point about just ripping the band aid off quickly and doing away with all of it.

[FrankJ]

The teams that are playing 4 and five event are likely influenced by being mostly district teams progressing to worlds or well funded regional teams. In either case likely high performing teams. It seems that would skew the graphs.

[D.Allred]

Jim,
Thanks for the analysis and interim proposal. I submitted the following comment in the survey since it seemed to lean toward an all-or-nothing approach.

You left out the option of out-of-bag time between competitions. At a minimum, we need time to practice and do some maintenance. Modifications and upgrades could still be done at competitions.

My hope was to lessen tendencies for teams to join the arms race by not eliminating regional withholding rules. Stop-build-day or something like it is still a good schedule milestone. Extra time and the hard stop of competition day will not make people better time managers.

In my opinion, teams will see most performance gains through practice and small improvements. Performing well with your existing robot will hopefully help with team retention.

Now we need a decent place for teams to practice…

David

P.S. Just to be clear to the other readers, I voted for no bagging requirements. It may simply be too large of a culture shock for FIRST. However, the teams need some type of relief. What other “sport” does not allow practice between events? "Sport for the Mind?"

[JB987]

Originally Posted by Joe Johnson

I am not seeing the disconnect that others see. I believe that I can be FOR removing the stop build rules and still be AGAINST having the top teams cheesecake the soul out of a lower tier team.

I have a yardstick. It has inspiration tick marks along its length. I take the controversial position that it is more inspirational to have a team compete with a working robot of their own creation that can accomplish a game objective they set out to achieve.

I don't like excessive cheesecaking because I believe it has bad long term effects on inspiration. I don't like stop build rules because they significantly disadvantage teams with low resources and while wasting resources of high resource teams, both of which adversely affect the inspirational impact of FIRST.

You can disagree with my views but I don't understand how they are incompatible views to have.

What am I missing?

Dr. Joe J.

+1

[Richard Wallace]

The Average Joes built a practice robot this year, because we knew (from our last two years) that the way to be competitive* is to practice. More access to our competition robot would have been a better solution.

IMHO, the issues of robot access and cheesecake can be separated. The motivation to offer or accept cheesecake is more dependent on game design.

I'm with Jim, Andy and Dr. Joe on robot access. How will we get there? I like Jim's proposal as a first step.

-------
*Like the way to Carnegie Hall, or the Olympics, or MIT.

[kaliken]

Originally Posted by Richard Wallace

The Average Joes built a practice robot this year, because we knew (from our last two years) that the way to be competitive* is to practice. More access to our competition robot would have been a better solution.

So I am curious. If you had no bag and tag would you still build a practice bot?

Actually for the entire CD would you still build a practice bot if there was no bag and tag? I just am curious in which way the community is leaning (especially from some of the powerhouse teams)

[AdamHeard]

Originally Posted by kaliken

So I am curious. If you had no bag and tag would you still build a practice bot?

Actually for the entire CD would you still build a practice bot if there was no bag and tag? I just am curious in which way the community is leaning (especially from some of the powerhouse teams)

We'd likely make one practice bot instead of two.

[Karthik]

Originally Posted by kaliken

So I am curious. If you had no bag and tag would you still build a practice bot?

Yes. For the following reasons:

1. The practice robot usually often serves as a "beta edition" of the competition robot. Developing one helps you catch issues that can be entirely circumvented in a clean manner on the competition robot.
2. There's only room for so many hands on a robot at one time. Having a practice robot allows for software and mechanical development to happen in parallel. Even with a true 16 week build season, there would still be time crunches where having to development platforms would be an asset.
3. Wear and tear. Our competition robot is usually at the end of its life span by the end of Championship. Our practice robot always in much worse shape than the competition robot. I can't imagine putting that many hours of drive time into one robot.

This probably doesn't apply for most teams. But for a team who has the resources to comfortably complete two robots, it's easy for me to see why they would continue to do so, even with the potential abandonment of the bag.

[ThaddeusMaximus]

Originally Posted by Karthik

Yes. For the following reasons:
3. Wear and tear. Our competition robot is usually at the end of its life span by the end of Championship. Our practice robot always in much worse shape than the competition robot. I can't imagine putting that many hours of drive time into one robot.

orrrrrrrrrr get rid of 120lb weight limit so parts can be designed for infinite fatigue life and still be viable?

[kaliken]

Originally Posted by Karthik

Yes. For the following reasons:

1. The practice robot usually often serves as a "beta edition" of the competition robot. Developing one helps you catch issues that can be entirely circumvented in a clean manner on the competition robot.
2. There's only room for so many hands on a robot at one time. Having a practice robot allows for software and mechanical development to happen in parallel. Even with a true 16 week build season, there would still be time crunches where having to development platforms would be an asset.
3. Wear and tear. Our competition robot is usually at the end of its life span by the end of Championship. Our practice robot always in much worse shape than the competition robot. I can't imagine putting that many hours of drive time into one robot.

This probably doesn't apply for most teams. But for a team who has the resources to comfortably complete two robots, it's easy for me to see why they would continue to do so, even with the potential abandonment of the bag.

Thanks Karthik,

You nailed exactly the reasons I stated in the survey on why we build a practice bot. We are very lucky that we have the resources to support building two robots. I know we have at least broached the thought about building three but we cannot justify this even though our programmers would love it.

As for us, we would most definitely continue building practice bots. I know that for us to maximize student involvement is to build two robots, many of our students are brought up to speed on building the practice bot. As a second reason, like industry (at least aerospace) we usually deliver an Engineering model that provides the same functionality to wring out the bugs through qualification testing. We do try to keep the students learning the engineering process typically with design reviews etc. (we are trying to teach something too!)

With the removal of bag day, I can see it being a very difficult decision for teams that are on the edge of capability on whether or not to build a practice bot. As cited above just having one adds dramatic value and in my opinion adds significantly to the overall competitiveness of the team.

I see removing bag day dramatically helping teams that have low resources upgrade their single bot, yet I still see an even tougher decision for those teams that are on the cusp. Basically in the end I still think a practice bot will be required at the upper echelons of play.

[Lil' Lavery]

Great working gathering and presenting this data. It was really an eye opening read in many ways.

However, I am given pause by some of the leaps taken when discussing the presented statistics in this paper. As engineers, I think we've all heard the oft repeated phrase "Correlation does not equal causation." There are some pretty dramatic leaps taken in the analysis of points 3 and 5 that ignore a host of other factors.

On point 3, to me Fig(2) isn't as clear as the proceeding paragraph claims it to be. The highest portions of the lost teams curve correspond with the high portions of the 2015 teams OPR distribution. That is to say, the most teams are lost from the OPR brackets that have the most teams total. That is obviously to be expected. Admittedly the skew shifts between the two plots, but I would like to see the actual loss ratios for each bucket rather than just raw totals.

Further still, while it's obvious from the tails of the plots that extremely poor performers are more likely to fail than extremely strong performers, there are a plethora of factors that could potentially explain that, rather than the teams failing because of their poor performance. Are these poor performing teams particularly inexperienced, underfunded, under resourced, or under mentored? FiM clearly has some degree of feedback on this, but the dynamic and culture of FiM varies greatly when compared to the rest of FRC given the levels of state sponsorship and funding. If FIRST HQ has similar surveying of teams lost to attrition, I would be very eager to see it. Given the other potential stressor on team retention among these extremely poor performers, I would be very cautious about making any leaps that a stronger on-field performance would result in them surviving to future seasons.

On point 5, I would like to echo the previous concern voiced by Greg Woelki. Each population in point 5 is a subset of the previous, but not a uniform sampling of the previous population. By removing 1-event teams from the 2nd even population, you're narrowing the sample to the teams that had the resources to compete twice and introducing a selection bias. There are even stronger selection biases with multiple event teams once you start factoring in teams that attended their district championships and/or FRC championship.

This selection bias is demonstrated in fig(6). Teams playing 1 event have a lower OPR at their first event than teams playing 2 events. That suggests that teams capable of competing multiple times are already at a higher level than those without the resources to compete multiple times. The upwards trends of all five groupings does mitigates the concerns of the selection bias to an extent, as it shows repeated plays do in fact help teams improve their performance, but the raw totals of the average OPRs mirror much of what is argued in point 6 (the better performing teams are already better and remain better). The average of the "Teams Playing 2" sample fails to reach the "Teams Playing 3" sample's beginning of season OPR, even after their 2nd event.

Most of all, both figures in point 3 are arguing that teams with more plays improve as the season progresses. There is a distinct difference between more plays (competition matches) and purely more robot access. While more competitions does mean more access, it also means a plethora of other factors, namely driver experience and competition field access. It's hard to say if more robot access alone would achieve the same levels of positive trends (or even if the gaps that already exist in point 6 could potentially be increased further). I'd be willing to wager that access to competition fields is a huge resource and a giant factor in the improved performance of teams that get repeat plays. I'd also argue that fig(6) even suggests this, as the steepest positive slopes in all four repeated play samples is between event 1 and event 2 (as teams get to test their robot on a real field for the first time).

Do not take this post to be a criticism of the concepts proposed in this paper or the elimination of bag day. Neither of those issues I have formed a strong opinion on to this point, as I see very valid arguments on both sides. Also do not take this as a criticism of Jim Zondag or the paper as a whole. I love the effort and dedicated to the program Jim has and the passion put into writing such a paper with the goal of moving FRC in a direction Jim feels is best for the program.

[GeeTwo]

Jim,
Thanks for all the great data analysis, tied together with great commentary! I have been a bit on the #keepthebag side, mostly from a "devil you know" philosophy. After a first quick read, I'm now squarely #onthefence, moving towards #banthebag.

Sean,
Thanks for pointing out all of the weak points I'd noticed as I read, and a couple more. Even given those, there can be no reasonable doubt that more time with hands on the robot and more drive practice (not necessarily in that order) means increased ability for a team to perform game functions and be competitive.

All,
Since reading about the poll this morning, I've been pondering the question of whether we'd still do a second robot if there is no bag, or (later), 8 hours of access per week.

As background info: we're competing in the regional model, and this part of the country is still several years away from the team density to support districts. For the foreseeable future, we're looking at district registration and full team travel and hotel costs for a second event. We managed to binge-fund a trip to CMP in 2015, and drew in a few more sponsors, but unless we get a mentor or student with a better talent (or at least drive) to draw funding, we'll probably be able to afford a second regional about the same time we transition to districts.

At 8 hours per week, we would probably expand our Saturday build (currently six hours) to eight or nine, and do a single unbagging each week where we did fabrication, drive practice, pit crew practice, and robot upgrades in a rush, and used our much shorter weekday evening schedules for planning, CADding, and working with a practice robot that we would definitely still build.
At 20-168 hours of unbag per week, the question becomes a bit murkier, but I still think we would do two robots. Two robots are already part of our pre-bag processes (swapping robots off between project groups, including chassis, manipulators, programming, and drive team), so unless we lose a significant amount of resources (which could be money, facilities, mentors, or students), we would probably tweak the second robot processes, but not cancel them.
The thing that excites me about a protracted unbagging each week is the possibility of a scrimmage. Currently, teams who do not build a second robot cannot even think about competing at a scrimmage, so there is no point in doing it in our area; I believe we are one of a very few. With an 8-hour unbagging window each week, I could definitely see enough teams to support a 3-6 hour scrimmage every week or two between "initial bag" and Bayou Regional, if we can identify a facility and carpet large enough to host the event.

[Siri]

Originally Posted by Lil' Lavery

[Figure 2]...I would like to see the actual loss ratios for each bucket rather than just raw totals.

[Figure 5]...By removing 1-event teams from the 2nd even population, you're narrowing the sample to the teams that had the resources to compete twice and introducing a selection bias. There are even stronger selection biases with multiple event teams once you start factoring in teams that attended their district championships and/or FRC championship.

[Figure 6] This selection bias is demonstrated in fig(6). Teams playing 1 event have a lower OPR at their first event than teams playing 2 events...

Agreed. I'd like to suggest a new multi-panel figure that shows the event-over-event OPR distribution separately for each population segment. So FigNa would be the 1st and 2nd event curves for 2-event teams, Nb is 3 curves for 3-event teams, etc. It may also help to normalize the y-axis by percentage of given population, and I'm sure there are ways to improve this suggestion with better cross-comparison. Still, it seems like we're making a big leap that we have (well, Jim has) the data to fill.

I'll also echo the desire for loss ratios by OPR bucket for Figure 2. The probably has a lot of noise, though, and if it's possible the case would likely be stronger by normalizing the OPRs and aggregating multiple years. I don't know what your database looks like though, so this might be a pain.

I think there's also a way to address the questions that arise with Figure 6, but I'm not sure what it is yet. There should be a way to directly handle the relative difference in OPR between the populations versus the changes in each over time (demonstrating the salience of each factor). Similar to what Sean mentioned, for 2- versus 3-event teams, the fact that you are a 3-event team appears to be almost as useful if not more so than actually playing your third event--I would guess largely because you're a team that's going to qualify for DCMP based on your prior performance (or CMP). This is not to dismiss the paper's Point 5 that the figure is supporting, but the data is interesting.

Overall, I think this case could benefit from talking more about the dataset. In OPR progression, how many DCMP and CMP performances are in Figure 5's green 3rd event line versus just being a 3rd "normal" (district or regional) event? Is there enough data from "normal" 3rd events to look at this directly, or do we have another proxy adjustment available? Dropping teams that didn't qualify for DCMP is certainly going to shift the OPR distribution regardless of play number.

[R2D2DOC]

All,

I may have missed this point in the great number of thoughtful responses: If the B&T is modified, what impacts/advantages can be realized for the competition season schedule?

Just Wonderin'

[Mike Schreiber]

Originally Posted by ThaddeusMaximus

orrrrrrrrrr get rid of 120lb weight limit so parts can be designed for infinite fatigue life and still be viable?

Not to get too off topic but interesting point

How do you plan to carry the robot to the field?

Originally Posted by The National Institute for Occupational Safety and Health (NIOSH)

...the maximum weight to be lifted with two hands, under ideal conditions, is 51 pounds

Source

FIRST already pushes this limit since we are usually lifting from the ground and load capacity is generally reduced to about 80-90% during a two person lift since the weight isn't exactly split by 50% at all times.