formal quality procedures in FRC?

[wireties]

Our team had a banner season but it was marred by hard-to-explain (and stomach) stretches where we suffered quality issues. At champs we actually put up 5 stacks of 6 on the practice field sandwiched by 2 Carson matches where we scored almost zero - very frustrating.

I think that FRC teams are intended to mimic companies (with some limitations). My reaction to similar quality issues in a company would be a formal quality program and an effort to make quality assurance a part of the "culture".

Do any teams out there have formal quality programs? Do they work? Might you share some details? How far do you take it? Do you have team members tasked with quality inspections? Do you inspect as you design and practice or only at competitions? Do you consciously design for testability and maintainability? In most companies, the quality folks seem to have a slightly adversarial relationship to engineering and manufacturing - do you take it that far? How might one incentivize students to pursue greater quality?

TIA

[GeeTwo]

I hope to get to that point in a few years - where we're seriously considering implementing a formal quality process. That doesn't mean I haven't thought about it, so I'll share these, with the understanding that I haven't actually tried to implement any of it.

Quality takes time. Any formal process takes time. And while FRC has definite dollar limits and weight limits and current limits and pressure limits and height limits and (most years) width and depth limits, development time is the most restrictive resource for the vast majority of teams. Ensure that whatever quality procedures you implement, that they are done in parallel with rather than in sequence with development. And don't forget that you can't QC end performance if you aren't performing prior to competition. This means driver practice, and in the best case, a practice robot that is essentially indistinguishable from the competition robot.

As far as quality issues as you described (irregular scoring), if you have a scouting department, put them to work gathering as much data at your practices as they do at competition. Not only is it good practice for them, but it will provide the info that there is a problem early, and should help to identify whether it is a machine, driver, or other problem. Along these lines, 3946 is planning to have a Game Analysis / Scouting / Strategy group next year, which will include the drive team. As self-scouting our drive practice is part of this group's function, perhaps we really are taking a step down the same road.

[wireties]

We built two bots and practiced quite a bit, more would have been better. And we changed the can grabber twice in mid-season - took time to get used to it. But our biggest mistakes in competitions were avoidable and sometimes embarrassing. I'm thinking designing the robot to be testable and then testing it with a rigorous checklist will help. But the culture part is tough...

[sanddrag]

On 696, we've won the Quality Award three times. Once in 2012, and twice this year. To us, quality is more of a mindset, a philosophy, and a way of doing things than it is a punch list. It's about carrying an attitude that "Good enough isn't" and "If we aren't going to do something as good as we possibly can, why are we doing it at all?" and "Every detail matters." Personally, I don't take part in things I can't do well, and I do pay attention to the details, because they do matter.

Quality is about recognizing when something is done poorly, and not being afraid to throw it out and redo it, no matter how much effort was put into it nor whose feelings may be hurt. Students have cried on our team during this process, but they now know quality, and are proud of their improved standards, and absolutely nail their tasks to perfection in many areas.

We try not to allow too much hack work (we lower the bar for prototyping), and we try to use professional grade software, equipment, and measuring tools to design, manufacture, and verify our parts. It's easier to make quality parts, with quality tools equipment.

Our vision to improve our team was heavily influenced by Team 968 and further inspired by visiting the San Diego regional in 2012 as spectators, and compiling enough photographs of teams' pit areas to make a PowerPoint presentation of "The Good, The Bad, and the Ugly." It has now become an annual routine viewing on our team, to show students the differences between teams' work habits, displays, and "curb appeal" if you will.

We also try to follow an industry-type workflow with our management of electronic data, using reliable servers for collaboration and long term storage of critical data, a check-in an check-out process for CAD data, and job-sheets for any parts to be manufactured on the CNC. Finally, this year, all sub-groups on our team were asked to develop and document a set of "Standard Operating Procedures" for everything that we do. Our goal is that one day these will be detailed enough for any new member to be able to do any role on the team simply by reading about it beforehand. The SOPs will eventually include everything from setting up for lunch in the parking lot of an event to operating a ratchet strap (mandated training for 2016) to fastener types to welding to manual and CNC lathe operations and more. This will be a multi-year effort, that I don't anticipate will be complete soon, but once it does reach a state of near-completion, it will be a fantastic resource for new members.

While most of our team can recognize when something is not as good as it can be, we've yet to have a formalize QA process. We really ought to soon.

[Tungrus]

Did you find out or try to find out why your robot could not score any points during a competition match? i can understand your frustration after the match with zero contribution to your alliance, we have been there and felt the pain. In our case it was not robot, robot was good on practice field and it performed as expected in many matches. We had taken extreme care to make sure our electrical and mechanical was sound. Sometimes the drivers' under lot of pressure deviate from the original strategy and when they mess up once, they will not be able to recover in 2 minute match. Sometime the drive team members discuss during a match, which causes them to lose focus. We talk to our drive team and calm them down, it is not easy.

As far as Quality, our robot is not the highest scoring one, but we do a good job inspecting everything during the build season as well as between matches. We make sure all fasteners are as they are supposed to be. We double check electrical connections (no dangling wires, tape them), tighten the terminal screws on motor controller (if any), make sure the circuit breakers/fuses are in place, the nuts on main circuit breaker are tight. The nuts/screws on the main circuit breaker and the motor controller are probably most ignored ones, and these things can get loose due to vibration. Also we make sure battery terminals are secured.

I am interested in seeing some formal Quality Control doc and how other teams deal with this.

[GeeTwo]

Originally Posted by sanddrag

While most of our team can recognize when something is not as good as it can be, we've yet to have a formalize QA process. We really ought to soon.

If it's in your culture, it doesn't need to be formalized - and formalization might just kill the goose that lays the golden eggs.

[wireties]

Very frustrating - every match was a different issue. We changed to a 6-stacker for champs from a regional-winning 5-stacker so the machine did drive differently. Our driver had a few problems (a few matches) but most problems were electrical and mechanical. We had a CANbus wiring problem (2 matches) then replaced a TalonSRX but did not properly wire it into the PDB (2 matches). We didn't properly position the cube robot, ours is an active tether design, one match. We tied a critical piece into a position to enable a 20pt auto effort by an alliance member but it did not release as expected (we were dead that match). We had joysticks swapped (one match) then had one flake out (frayed cable) - just one silly problem after another. Arghhh!

[wireties]

Originally Posted by GeeTwo

If it's in your culture, it doesn't need to be formalized - and formalization might just kill the goose that lays the golden eggs.

I can see that might be the case on a FRC team. But a company (that makes machines, not web-ish stuff) would never leave anything to chance, there would be a parallel formal QA effort.

[GeeTwo]

Originally Posted by wireties

Very frustrating .. most problems were electrical and mechanical. We had a CANbus wiring problem .. just one silly problem after another. Arghhh!

This sounds a lot like our Ultimate Ascent experience. That was the last year that we had programming and controls as part of the same department; our controls lieutenant described the wiring situation as being like "4-d TETRIS". We intentionally split programming and controls the next year so wiring and pneumatics routing would have people speaking up for it. We also managed to bring in Isaac, a former-IBEW member as a mentor. This was luck rather than intention; his son Seth was interested in programming. Even after Seth left, Isaac stuck around and has become a key mentor on both the technical and business sides. Check your local IBEW hall for mentors!

In any case, it sounds like a number of your problems this year could have been solved, or at least better managed, with checklists. We have been increasing in our use of checklists each year. As Barry, one of our newest coaches/mentors noted at our first post-CMP meeting - "We've been using the wrong sports model. FRC is more like NASCAR [than it is like football, baseball, basketball, or tennis]. And the guys in the NASCAR pit don't [relieve their bladders] without a checklist."

[Brian Maher]

After being a victim of a variety of mishaps in previous years, Team 1257 has implemented a pre-match checklist that is completed before every match (practice, qualification, and playoff), listing vital maintenance and match switching procedures. One Pit team member is designated "Inspector", and it is his/her job to oversee the checklist's completion. It has made match turnarounds much more thorough and smooth, and reduces the chance of mistakes.

We also keep a log of technical failures that occur during competition. This helps us notice patterns of error and better address them.

[Qbot2640]

As a quality engineer by trade, I have to admit some embarrassment at not thinking harder about implementing a quality focus toward our FRC team. After reading through this thread, it occurs to me that the FMEA (Failure Mode Effects Analysis) could be particularly useful here. In the early days of build season, the DFMEA (design) to consider all the possible things that could fail with the design and what effect they would have. This could be an effective way to determine where a flaw in the design could lead to a problem you would not otherwise have discovered. As strategy is developed, and procedures are developed for event preparation and participation, the PFMEA (process) to figure out what problems could occur surrounding team performance - similarly useful to guide strategy, maintenance, and other competition considerations.

[IKE]

I was working with 469 this year, and they have a couple practices that are worth sharing:

#1: system Checks in the pits. This is a recommended set of checks for each subsystem to know that the robot is ready for its next match. I highly recommend making a set of checks and refining the list as your season continues.

#2: During build, they ahve a really cool system. The students design every part in CAD that must be fabricated. They then make a drawing for each part, and have a part number scheme that starts with 3 letters that represent the subsystem. IE Arm parts are: ARM001, ARM002.... Stacker parts were: STK001, STK002.... A drawing goes out with each student fabricating the part. They then use 4 colors of stickered dots to code each part. One for in process comp bot & 1 for in process practice. One for Approved comp bot, one for approved practice bot. As soon as the rough stock is cut for a component, it is labeled with the "in-process" sticker which has its PN on it (IE ARM001-yellow). This sticker stays on until the part is "complete" and it is reviewed by the student design lead who approves it by removing the in process, and replacing it with an approved sticker (IE ARM001-yellow).
They keep all the fab parts on a set of shelves with a label for each level corresponding to a subsystem.
They also keep a Fabrication BOM that has all the parts organized by subsystem, and the same color code with a 5th color for not started. They added a column for "milling required" in order to help prioritize/schedule parts through the mill which was one of the limited resources this year.

The Fabrication BOM was a great tool for helping the build coordinator(mentor) know what was ready and set daily priorities as well as guage readiness to plan and add shop hours as needed.

This was my first year dealing with such a slick system (33 utilized a different system that worked well, but differently when I was there). A key component was the team was pretty disciplined to utilize it. This supported a very fast assembly schedule which allowed for some additional design time compared to other teams I have worked with. It also ensured that practice robot was very very similar to competition robot. This allowed for tweaks to be tested on practice and then later implemented on comp bot during the season.

One drawback with this system was practice robot was not "done" (or more appropriately labeled "useful") nearly as early in the build season as some other teams I have talked to. I don't think practice robot was terribly useful until the last couple days of build vs. some teams have a useful "practice" robot around week 4.

************************************************** *********
"Formal Quality" is a pretty broad topic. At my work, it encompasses a lot of different areas, and items. Occasionally, it can be adversarial, though that is often due to "design expectation" not meeting design need. This can be especially true in a "protoype" environment.
************************************************** *********
To the original thread starter, some recommended reading is:

Checklist Manifesto. How to get things right.

Atwul Gwande was part of the World Health Organization responsible for the Surgery Checklist. The results of utilizing a checklist were amazing:

http://www.nejm.org/doi/full/10.1056/NEJMsa0810119

From the above link:
"Use of the WHO Surgery Checklist reduced the rate of deaths and surgical complications by more than one-third across all eight pilot hospitals. The rate of major inpatient complications dropped from 11% to 7%, and the inpatient death rate following major operations fell from 1.5% to 0.8%."

As a reminder, these are some of the most educated people in the world conducting these procedures, and reduction of in-hospital deaths between 30-50%

Because it is a small number, you might say, so what. There are a lot of oeprations in the world. Also from the link:

"Why is the Checklist important?

Surgery can be a life-saving or life-changing intervention in many conditions and the provision of surgical services is being increasingly recognized as a significant public health issue. A modeling study estimated that 234 million operations are carried out every year across the world. This translates to one operation for every 25 people and is more than the number of children born worldwide each year.
An estimation of the global volume of surgery: a modelling strategy based on available data
Lancet 2008

However, despite the positive impact the provision of surgical services can have on a population’s health, surgery itself carries risk. Current estimates of morbidity and mortality following surgery indicate that over 7 million people worldwide will suffer complications following surgery. One million of these people will die as a result. Around half of these complications are potentially preventable, so using the Checklist to improve the safety of surgery will save many thousands of lives each year."

Now, using the checklist and associated practices could/should in theory save: between 300,000 to 500,000 people per year! To put that into context. I believe there were around 40,000 people at the FIRST World Championship and Festival. Imagine saving 8X to 12X the number of participants... every year.

How does this apply to robotics?

A lot of robots end up having issues during matches that keep them from working. I am not sure the exact ratio, but I would guess somewhere around 10% or about 1 issue every other match. This seems a bit low, as I know a lot of very good teams tend to have at least 1 issue per event (with a 12 match qual schedule). Several of the Greats are able to keep a clean schedule at most events, but that is a delta from "really good" to "Great". Often 1 qual match victory can be the difference between #1 and #4/5 seed.
So, if you can reduce system failures from 3 per season to say 1 or 2... you may have a dramatically better season.

[wireties]

Lots of great info here! Thanks! I'm thinking we'll have formal in-process inspection by mentors, hopefully quickly handed over to students, and a rigorous checklist at competitions by students. We'll write up the plan in next few months.

[mathking]

I think there could be two different concepts lurking here. One is the physical quality of the robot. We have found (after years of competition and many unforced errors) that checklists are quite valuable. We have three basic ones we use.

One is the setting up the robot on the field checklist, which includes things like turn the robot on, make sure the robot is properly positioned, make sure the safety bar is removed (that last item came in handy a couple of times last year), and "make sure the proper auto mode is selected."

The next checklist is the one for when the robot arrives back in the pit after a match or practice. It includes take out and charge the battery, put in a practice battery, check the chains (if any), visually inspect the pwm and power connections, and checks for specific problem areas.

The final checklist is the before heading out for the next match list. It includes things like correct battery is in the robot, any prepositioning of components is done, in bumper years the correct color bumpers are on, this year the correct LED light color is selected, driver station is ready to go and the like.

These don't eliminate unforced errors, but they minimize them. (For example, in 2012 we did not initially have "balls loaded" on the checklist. We learned.)

The other concept revolves around how the match itself is actually played. Mistakes by the drive team, interference by allies, defense by opponents and other things can have an effect on how the match turns out. One important thing we try to do is to talk about each match immediately after the match. Video review can help with this. The view from the stands is generally far superior to the view from the driver station, so discussing each match immediately after can be really useful. Be sure to keep the tone of the discussion constructive and not accusatory. Constructive after action conferences at the conclusion of each match can be very helpful in identifying and implementing good practices in competition.

We generally try to divvy up the responsibilities of the drive team so that someone is thinking about what is going on now and someone is thinking about what to do next. Driving in a match is very different from driving on a practice field. There is far more going on in a match, and it is consequently much more difficult to focus on the task at hand. One thing we try to focus on is how to most quickly and effectively communicate so that the drivers know what to do. This reduces the incidence of "paralysis" on the field when the robot is sitting as the humans try to decide what to do. A good coach can help a lot with this. It is one of the reasons we have generally decided to have a mentor coach rather than a student coach. Our on field coach is very good, particularly at staying calm and focused. He generally finds that the more we play, the less he is instructing the drivers in what to do and the more he is looking around to keep ahead of events on the field.

[evanperryg]

Originally Posted by wireties

Very frustrating - every match was a different issue. We changed to a 6-stacker for champs from a regional-winning 5-stacker so the machine did drive differently. Our driver had a few problems (a few matches) but most problems were electrical and mechanical. We had a CANbus wiring problem (2 matches) then replaced a TalonSRX but did not properly wire it into the PDB (2 matches). We didn't properly position the cube robot, ours is an active tether design, one match. We tied a critical piece into a position to enable a 20pt auto effort by an alliance member but it did not release as expected (we were dead that match). We had joysticks swapped (one match) then had one flake out (frayed cable) - just one silly problem after another. Arghhh!

I don't know how many matches "a few" is, but if your driver is messing up in more than 25% of your matches, you need a new driver. The #1 most important aspect of a driver is their consistency behind the glass. If you have one driver that can put up 100 points 50% of the time and another who puts up 80 points 100% of the time, you should always go with the second driver.

Whenever you make any changes to your electrical system, have at least two knowledgeable, experienced students check your work. This will ensure any mistakes are found. After every match, check your electrical system, especially places that are high-risk or would cause serious problems if they broke. (For example, CAN lines, Ethernet cables, pneumatics)

Sounds like some of it is just bad luck, but some of those problems are preventable. However, you don't need any kind of documented policy to have a reliable robot. Make the pre-match systems check a part of your team's culture. Attention to detail should be practiced in the pit, on the field, and in everything you do.