Design Review to Mentors

Competition Team Organization
1

An idea that has been thrown around 103’s mentor group for a little while now is professional design reviews. The idea is once the students (read engineers) decide what their strategy is, what they want on their robot (in terms of mechanisms, drivetrain, etc), and the general layout of said mechanisms, they make a quick presentation to the mentors (read managers) and the mentors give constructive feedback on the decisions that were made and how they can possibly improve them. Then later in the season we can do a similar thing with the CAD design of the robot and the mentors can suggest ways to make manufacturing easier or to make the mechanisms more reliable/durable/functional/whatever.

Ideally the students would take a day or so to create a quick presentation with whatever napkin, back of hand, and notebook sketches they have and see what the mentors have to say. Then a similar review of the robot CAD before we really start to manufacture anything.

Not only will the design reviews give the mentors a chance to give their input (because we are typically very hands-off), but it would give the students a different perspective on their solutions and also improve the communication between students. It also emulates what a real engineering team would have to do on a large group project.

I am sure many other teams do something like this and I wanted to hear if it was an effective way to give criticism, improve the quality of the final product, and inspire the students to improve their solutions.

So CD, do design reviews work for you?

2

We had a very large group of students, and relatively thin leadership in 2015, so we took this approach, doing dry erase or similar sketches and some cardboard models, several rounds on kickoff day. Students did the ideas, mentors did the feedback. We had a high-level robot concept by the end of the day. Though we took this robot to CMP, we really missed some key points on the scoring and strategy, and felt we did not pass on as much to the students as we could have.

Since then, we’ve spent Saturday (and our only Sunday session of the year, in the afternoon) working on a strategy (not a design), and let the design work like this for about a week, with reviews each build day (Monday, Tuesday, Thursday, Saturday). We still use this general approach, but the designs are a bit better developed and briefed to everyone, mentors and students alike. The mentors let the students go first with observations, questions, concerns, and recommendations. Next, we try to ask leading questions and raise concerns which will help the students to find the problems. If these fail to raise the concerns we have, we’ll just dive right in with the critique and suggestions. We find that this process helps the students learn the critical evaluation skills as well as improving the robot design.

3

We find design reviews to be really helpful. They make sure everybody is on the same page. Our team has done Strategic Concept Reviews (whether they realize it or not) and Critical Design Reviews. In the past, we’ve invited our waterjet sponsor to come see it (design reviews are a good way to show your manufacturing sponsors and interested parents what you’re up to).

Here is a presentation we put together for our CDR in 2016: https://drive.google.com/open?id=14c3M7jgcswl4JnPMHoLGoI6QNjOfDL3WWzxT9ZcpLuA

That was the first year we had done design reviews and was more formal than we preferred. Last year, I think we just wrote things on large sheets of paper and stuck them up on our marker board. We’re still searching for the right format of design review for our team.

4

I can attest to their effectiveness. We had a big design review during our build season when we wanted to transition from prototypes and conceptual CAD to design and fabricating the final product.
You can read more about the details here on page 43 of the pdf.

Anyways, beyond just mentors, we also invited representatives from our sponsors. We had reps from UTC and Boston Scientific.
We would also have smaller reviews throughout the build season as well, but not all the mentors were in attendance and representatives from sponsors were generally not in attendance. But, reviews, even if it isn’t with a mentor panel, are very important. They provide the opportunity to gauge progress and refocus.

The structure of the meetings was generally as follows
-Prep: We feel that whatever progress we had made is ready for a decision or transition to the next step. Then we set up the meeting time
-Present: Whatever we had worked on, whether it was strategy, prototypes, or CAD was presented
-Questions/concerns: Mentors (and sponsor reps if there) would provide feedback for what we had done. We would answer the questions, address concerns, or discuss how the concerns could be addressed if our design had a potential flaw
-Decide: With input from the mentors and sponsors, we would walk out of the meeting with a direction for the future. For example, we decided on what prototypes would be implemented in the design and the plan of action to do that effectively, or what things we needed to change on potentially flawed designs

The reviews benefited the team in a few ways:

-Concerns and Questions: The review provided a time for the mentors and industry professionals to bring up their questions and concerns regarded mechanisms or designs. They may catch something us students missed when designing the prototypes or robot. As reviewing designs or ideas is common in any profession, it gave us students an experience to have a long discussion about the proposed project/product.

-Provided structure: In years previous, we would argue for an unnecessary amount of time and days on whether we should go with design a or b. This review provided an structured avenue to solve this issue: it was going to be resolved in that meeting. With the mentors’ and sponsors’ help, we were going to come out of the meeting with specific goals set, like which designs we were going to pursue. This made our build season far more timely than in the past.

-Sponsor Relations: This is a great way to keep positive contact between the team and sponsors. We even had one rep come back occasionally for the rest of the season to see how we had progressed. This could be a way to enlist another mentor or support in some other fashion.

One warning: They can become very long. Our big one was upwards to four hours. Try to have a set amount of time allotted to the review and stick to a schedule, transitioning when need be.

5

1676 has done these for years, to good effect.

It’s also a good way to get local business and community members involved, because it’s a three hour commitment once a year. Good way to interest sponsors as well

6

Absolutely. Especially if you’re still looking at more than one design, appoint a timekeeper/moderator/sgt at arms and schedule the reviews on each design, holding some time out at the end for decisions and comparisons. Make sure that you spend more time prototyping and creating designs than you do evaluating them and knocking them down. We try to keep this time (involving the whole team) down to about 1/3 of our meeting hours the first week, and significantly less as the build progresses.

7

The overwhelming majority of what I am hearing is that this is a great thing that teams gain very valuable experience with.

Interesting to have design reviews each day, almost seems like too much oversight to me. Maybe I am a bit sadistic, but I like to see the students struggle a bit more :p.

But in all seriousness, this seems like a reasonable way to go about it, just very different from what they might see in the real world.

Thanks for the link to the presentation, and that is a very, very interesting idea to bring sponsors in to present the design review to. How much do your sponsors get involved in the rest of your build season? Ours are VERY hands off (they show up once maybe during build season, and maybe come to a competition).

Glad you pointed this out. Will try to do our best to keep within an allotted time frame and not waste too much time talking and not enough time doing.

Again with bringing sponsors in, this seems like an amazing way to give sponsors tangible effects on the teams, and a great way for students to interact with sponsors.

Like I asked before, our sponsors are typically very, very hands off. I am not sure they would even be interested in something like this, but it may be worth a try.

Thanks for all of the suggestions!

8

Now that I am on my 9th full time job after college, I have noticed a correlation between the quality of the design and the rigorousness and thoroughness of the design reviews conducted at each company. The rigor of the design reviews also correlated with the constructiveness of the (friendly) feedback given.

I think that knowing that one would have to pass such a design review meant that the engineers were more thorough and conscientious in trying to find any flaws in their own work before submitting it for review. I observed very little “throwing of mud against the wall” at these companies.

I am hoping to institute a short, weekly progress/design review this coming season, starting with off-season project. I am still the only technical mentor with the team and about 16 of the 20 students are total rookies to robotics.

9

696 used to do this way back in the day, but we haven’t for the past several years in which we’ve been pretty good at FIRST Robotics. We ditched this process when we realized that we spent essentially the entire first week designing many different robots and ultimately ending up right back at square one with nothing decided, when the goal is to just design one robot. So now, we focus our efforts on designing one effective robot, rather than splitting the team to design every conceivable robot.

10

On my high school team we did something similar 3138, but not as detailed. It was ok, we had other problems with agreeing on which direction to go in.

With the current team I mentor, things have gotten better as students and mentors have been in a tighter loop of communication. I’d say design reviews are good, but having a tighter design review cycle is ideal. Part of the challenge is to get the students in a state where they can easily ask for advice, and not have them spend time going down a path that isn’t promising.

Using grabCAD has been useful, and has become more useful as students are more eager to get feedback immediately. It makes reviewing the content much easier as you can do it online, and comment publicly. Also, when students are in the habit of uploading at the end of each day, it’s significantly easier to catch issues sooner. Lastly, different subsystems can see what is being worked on, and when it comes time to present ideas to the team it’s great.

Another habit that has helped is weekly leads meetings with all mentors present and everybody shares progress.

Lastly, slack has made a significant impact on how tight team communication is. It helps tighten communication which has helped students be more constantly in contact with mentors. A really nice thing during the beginning of the season is sharing videos of every prototype that is made on slack, this way everyone can stay in the loop and informally discuss the direction of the team.

11

We do not “design every conceivable robot”. We do not begin the (serious) design until we have worked through our strategy. The various designs at we do through week 1 are figuring out what (for example this year) our climber will be like. We had a bunch of different ideas for how to secure the rope, some worked better than others. By the end of the week, it wasn’t just decided, it was obvious how we were going to do it.

Doing a daily mini-review definitely helps us decide better which designs to stop pursuing earlier than if we waited until the end of the week. I had an Admiral tell me just yesterday that the essence of strategy is deciding what **not **to do.

12

Thanks for all of your input everyone.

This is one of the main side advantages I see with this kind of review. If the students know they have to impress the mentors/sponsors, they will likely do a better job of double checking their designs and ensuring basic things would work. The end result of the second (maybe third) design review would hopefully make the manufacturing of the bot significantly better.

Thanks for your input sanddrag. Maybe this was a problem with the design process and not the design review though. Our team usually comes up with the “final design” (which does change slightly throughout the build) within the first 3 or 4 days.

We are going to be using Weighted Decision Matrices this year to try and combat the “Design all versions” problem we have had in the past. I HIGHLY recommend looking at these to use on your team to come to a conclusion faster, and with more confidence you will be picking the right solution.

The grabCAD idea is absolutely awesome for design review and such. I imagine keeping a revision system would allow you to see the progression of the design over time too (which would be an AWESOME learning tool). Having it be public to the entire team for the entirety of It would be even better if the comments could be anonymous.

13

When it comes to robot design, we essentially do mini-reviews constantly. Since we don’t do everything in CAD first, so many of the details get figured out as we build. We have larger biweekly reviews with just the design team, helping to ensure all the different parts are on track and helping to guide them towards how they should be communicating between themselves.

We have done more specific larger reviews on occasion. The best one I can remember was a 2013 code review. Students walked us through the entire code base (Between our two regionals), and we managed to finally identiry a race condition that was causing some issues with the PID control of our climber. It was a really neat and valuable experience, I think.

14

All great points. Our team does just about everything we can to develop concepts fast and then filter them down to more quality solutions as soon as possible. Though we have used pure democracy in the past (which, in my opinion, has no place in any kind of engineering design), we will be implementing Weighted Decision Matrices this year.

15

The more and more I am hearing from other teams the more I believe this would be an excellent way to help our students improve their designs and critical thinking skills.

I didn’t even think of the communication part, but I definitely see how keeping everyone in the loop during a design review could be incredibly helpful.

16

You are also less likely to have to ask the team members to read the rules and it will reduce the number of times you will have to explain that a quadcopter is not feasible with the power sources and motors available and likely to be ruled too hazardous to pass inspection.

This technique is very good for geting emotions out of the decision making process so that the decision is based on merit instead of inevitable personal biases. Sometimes, the resulting decision is counter-intuitive and, upon reflection, proves to be superior to the decision that “gut feel” would have produced. It also helps eliminate “voting for the option my friend voted for because I don’t know which option is really better”. We have taught this technique to several FLL teams in the area and they have put it to good effect. One won the Regional Championship 3 years in a row. The team that was in second place two years in a row used the same techniques.

17

We don’t do this in a formal capacity as often as, perhaps, we should - but we get the student leaders together about once a week during build season and have them explain the state of the robot and their current plans. During this time, usually we comb through several CADs in various states of completion, and make liberal use of a whiteboard.

What I’ve found to be even more helpful, though, is to have regular updates posted to a team management software system such as Basecamp. This provides a natural way to maintain ongoing design review and dialogue. It is hard to overstate how much of a difference this has made for our team.

18

This is also an excellent place for a team member who may not be so interested in technology and “doesn’t know how to build robots” but has a flair for organization and management. One of the teams that works out of the same facility has had a team captain who “did not touch the robot” and spent most of his time organizing and coordinating the teams efforts. After confirming that he had some natural abilities for this type of work and that he enjoyed it, he went on to pursue a technical management degree in college.

19

We do this. It helps a lot with team buy-in to the robot design. It’s good practice for everyone as well, even mentors who present things for review.

We have a few rules about freed-back and presenting to keep things going well:

  • Never, ever, ever, ever, ever, ever start a sentence with “why don’t you…” because that only invites negative commentary and often positive merits are completely ignored
  • Avoid “what if we…” because it shows that a comment has little thought behind it as to its implications
    -Avoid using I, You, or “…'s idea” because FRC is a team project, not an individual project. It is important to reinforce that we’re in this as a team, and it helps avoid criticisms turning personal, i.e. “bob’s idea sucks because…”
    -Use We, Us, Our, or Our Team, this reinforces a team mindset and depersonalizes criticisms
    -Ideas should be presented like: “We should consider … because … will support our chosen strategy!” This shows that an idea has been thought through a bit and has some level of merit.

We all think and interact in language, so the quality of our thoughts and interactions are only as good as the quality of our language (paraphrased from George Carlin). These simple rules have really helped us have productive discussions.

20

I can’t say I agree with these.

“Why don’t you…” is a very useful phrasing, because it provides the person presenting their design with an opportunity to provide their justification for a specific design choice in the face of an alternative. Simply stating “we should consider doing…” does not do this - when presenting a design, I would much rather be asked why I decided to design something the way I did rather than told that something that I may have already considered and discarded should be considered. The former is reasonable, the latter is, arguably, belittling. Moreover, the answer to “why don’t you…” is occasionally “I didn’t think of that,” and that exchange is often one of the most productive, in my experience. None of this needs to be negative - if your team is finding that it leads to negativity, I might suggest that there’s a deeper problem that needs addressing than the specific wording.

Additionally, I think that stifling “off the top of one’s head” ideas during a design review (“what if we…”) would severely decrease the quality of many of my team’s final designs. If an ultimately unsuitable idea is briefly considered, we lose a small amount of time. If a superior idea remains in someone’s head because they were afraid to share it for fear of it not being bulletproof, we lose potential functionality. Unless we’re struggling with our meetings being derailed by chasing too many tangential design suggestions (we rarely find this to be the case), we’d rather err on the side of “get it all out there.”