Pilot Year Review: New Chairman's Award Feedback Structure
 

Now that we've had a full season of Chairman's submissions, and experienced the full changes of the pilot feedback system, what are your thoughts? The new setup now includes:

• Regional and District Championship winners have their essay/executive summary and chairman’s video posted online. It was strongly encouraged that winning teams submit a practice presentation video as well, though not every team took part in that.
• The team is allowed an extra two minutes to present, if they so choose (5-7 minutes), though the 10 minuted total limit stayed the same.
• A mentor was allowed to sit in on the presentation session without counting against the 3-presenter limit. The mentor was theoretically allowed to film the session, but from what I understand that wasn’t true at all events.
• No formal feedback from the judges at any event.

So, what are your thoughts? Do you like it? Do you hate it? Was it as bad as you thought it was going to be? Were there unexpected benefits? And what direction do you think FIRST should consider moving next year with this?

Previous discussion about the decision when it was announced here: http://www.chiefdelphi.com/forums/sh...d.php?t=133343

FRC Blog post announcing the change here: http://www.usfirst.org/roboticsprogr...award-feedback

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I very much disliked not getting a feedback form. It's the only real way to know how to improve your team. Having a mentor in the room only helps you get better with your presentation skills, it doesn't help the team as a whole move forward. How are we supposed to continue moving forward as a community to change the world if we don't know how to improve ourselves?

All of the other changes were excellent. No problems with any of those.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

This was nice, and it'll definitely help teams in the future.

This is perfectly acceptable and just plain makes sense to not have 5 minutes be a super-hard limit.

This is a good change, although I heard absolutely nothing all year from any source about the mentor being allowed to film the session.

I still just don't understand this. There's absolutely no legitimate way for a team to gauge how a judge ACTUALLY felt the team did on not just their presentation, but their essay as well. Getting to see who 'beat' (in quotes because FIRST is trying to push not viewing Chairman's as a competition...) you is not an acceptable form of feedback at all. It doesn't tell you what your team's weakness was; and gives nothing for a team to build on in order to improve at presenting their material.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

After giving it a go this season and discussion with the team we preferred the feedback forms, mainly because we didn't use the mentor-in-room for a number of reasons.

We realized before the presentation that having a mentor observe would only add an "harder" step of guessing:

2014: Present -> Feedback Form -> Attempt to discern what more the judges wanted -> Improve presentation for next regional/year
2015: Present -> Mentor listens -> Mentor attempts to discern how presentation/answers could improve by trying to "read judges" -> Improve presentation for next regional/year

Most of the improvements a mentor would see outside of "reading the judges" would be presentation flow, answers to questions, etc that we had been practicing and working on for weeks already (in our opinions). There wasn't a lot there that would become "newly apparent" to us. I can't imagine a situation where the mentor would be able to glean more than the students in how the presentation/answers could be improved, but then again we didn't send a mentor in.

Between the 2 reasons above, concerns about the viewing mentor being able to keep a straight/neutral face through the presentation (if a line were missed or question were flubbed) as to not affect it, and other assorted issues, we decided to let the presenters go in the room and do their thing without the additional pressure.

I think they could do away with the giant however-many category rubric (those never helped us anyway) and just have a few short written feedback questions like the ones in previous years..

Flexible presentation/question time was good. We aimed to have max time available for Q&A, but not having to hit 5:00 on the relative dot was much better for us this year.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

Us at Chop Shop dislike the lack of feedback. An outsider's perspective on what is working and what isn't is great to have.

More often than not, it really helps knowing where we can focus our efforts in the off-season or if we're just totally missing the mark all together.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

Most people that I've spoken to have pointed out the lack of feedback as a negative, and I have to agree, for many reasons. That being said, I attended the Chairman's Chat presentation at Championship and the same issue was brought up. Karthik responded that the Hall of Fame teams spoke to FIRST about the issue when the decision was first made and they were surprised that it was being considered an issue, as only one team besides the aforementioned Hall of Fame teams had spoken out by contacting FIRST itself.

When we take issue with a policy that FIRST implements, it is our duty as teams to speak out. Clearly, we all thought that everyone else was going to send an email about it and didn't bother doing it ourselves. The clear solution here is to voice our passion to FIRST. They aren't going to fix it if they don't know that it's a problem.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

That's interesting. I know on my part, I wasn't aware of the changes until they were already implemented as policy. Looks like I need to keep a closer eye on what's happening at an organizational level.

In that vein, it may not be a bad idea to submit an organized grouping of feedback from teams regarding the policy change after experiencing it all the way through the season.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I love being able to read all of the essays and executive summaries that teams submit! I would really like to be able to see more of the winning teams' presentations.

This change did not effect us all that much. However, I really like having the breathing room.

I believe that this was a neutral change, I don't really have a reason as to why I liked it or why I didn't like it. It really depends on what individual teams want, I know some teams who did not want a mentor in the room and other teams who did.

I have no words for how disappointed I continue to be over feedback not being provided. At both the FTC and FRC levels, I find that feedback is key to improving teams around the world.

We have the ability to give feedback to FIRST on all of their events, we get surveys on every event we attend. So, as a team, why does FIRST not provide feedback on how to make my team better?

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I question this. I know for a fact I contacted FIRST (in my capacities as a Judge Advisor, Planning Committee member, and mentor) and I find it very hard to believe we were the only ones.

It's also completely at odds with the feedback I got from Frank in which he was well aware that the lack of feedback was a problem (I received this feedback two distinct times, during build a week after the announcement was made and during a feedback session at Championships)

In fact, I know that FIRST is being pushed to provide even more feedback. DL and Entrepreneurship were the top of my list but I'd like to see some form of feedback about the interviews conducted by judges. HQ has been made aware of this but there are a ton of logistical issues with implementing more feedback to teams.

I'll tell you the same feedback I've said all along, it's a step backwards. I understand why the step was made but I hope that we can come up with a solution to those problems and give teams back their feedback.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

• Love it. It encourages accountability in the submission, since teams know that the community can and will read their submissions, and it's a great resource for less experienced Chairman's teams. This is pretty clearly a great change.
• The extra two minutes was great for us - we have a lot to say, and it gave us a little more time to say it. Nice to have more space to convey your message. Less Q&A could sometimes hurt though.
• I'll combine these last two points. I'm very opposed to these two changes, especially since the first was used to justify the second. Here's my take: our head mentor attended all three presentations that we gave, at the district, district championship, and championship level. He's awesome - the team wouldn't exist without him, and he's been a valuable source of feedback for the presentation team all year. That being said, my presentation partners and I know the presentation like the backs of our hands, and as a result, by the time we present, the amount of presentation feedback that we need is minimal. The feedback that we need is feedback from the judges, outlining our strengths and weaknesses from their perspectives, and since our mentor, as great as he is, is not a judge, we cannot get this kind of feedback with the current system. That's all there is to it - mentors provide great presentation feedback, but since they aren't judges, they can't give us judge-specific feedback, which is what we need in order to improve.

I believe that if as many teams as possible contact HQ and tell them about this, we can get feedback forms back for next year. So don't just stand there - email FIRST!

To sum things up:

• Great
• Great
• Needs to change

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

The current method is ridiculous. A team mentor is going to focus on how the presentation was presented, not the content of said presentation.
If I had it to do over again, I'd have a mentor from a friendly but separate team sit in our Chairman's presentation, and offer to do the same for their team.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I question this as well. I know that my team contacted FIRST about this and begged them to reconsider.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I've actually not found the feedback form to be very useful in previous years, but it was something...and I don't like that it was removed. I agree with those above who point out that they want feedback about the entire effort, not just the presentation. Prior to this year, our submissions have been rather weak, thus the usefulness of the feedback form may have something to do with the judges not wishing to be overly critical...I wish they would have been, though.

I planned to view my team's presentation at Palmetto just to offer an additional perspective while they honed the presentation for North Carolina - but the judges were not informed that my presence was permitted in addition to the team of three, thus it would have cost us one of the presentation team members. So we essentially received no feedback beyond the perceptions of the presenters themselves...which we always get. That part of it ended up working out for the better, because our presentation team completely re-did the presentation on their own anyway, and it was so creative and awesome that it actually brought tears to my eyes when they told me what they did!

Summary - (1) bring back an enhanced feedback form that actually describes your strengths and weaknesses, and possibly even benchmarks where the judges considered your "performance" on a continuum that includes the "performance" of the winning team. (2) draft a single overall summary document and copy to all the teams that participated in chairman's with best practices seen and with observations that could use improvement. (3) Communicate rule changes to the judges better, so we all get to take advantage of these kinds of opportunities.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

This is by far the best change. It's extremely eye-opening to see what we can try to achieve.

We tend to talk a lot, so this was definitely useful.

I was initially very into this, but then started to think I may be putting too much pressure or may cause an issue, so I left it up to the students if I should sit in there or not. We decided that it would ultimately help. However, at this point I already had a pretty good understanding that our content wasn't as strong as potential teams because of the first change.

From my POV: Our presentation was on point from what we practiced. It was by far the best presentation we gave. The questions were pretty good feedback though, it pointed out a lot of things that we need to add on (as far as content) goes. I could see the judges start struggling to ask questions, which (being a judge before) is a decent indicator to me that we need deeper content.

I don't really know how to feel. I've been a big advocate of formal feedback for FTC, and I see the other side of the discussion. It sucks because a lot of pressure is put upon the Judge Advisor or Person-In-Charge, to field any issues or concerns that may come up. It only takes one bad apple (team or mentor) to yell at a Person-In-Charge for this to happen (which I'm sure is what happened).

I'm not sure how Chairman's judging works, but as a judge it takes a lot of extra effort to write down really really good feedback. IT's got to be constructive, highlight the good aspects of the team and provide suggestions for improvement. In the course of a competition....this is a lot of information to do for 30 teams. I cannot imagine having to do this perfectly for 50 to 60 teams at a FRC competition.

So...personally, I was ok with this for this year. We weren't very competitive and I could take away what the judges were looking for.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I like the soft limit on the presentation. I like posting the winners' submissions. I like being able to have a non-speaking mentor in the room to observe. I am very disappointed by the removal of the feedback form.

I was our non-speaking mentor for our first Chairman's presentation and our Dean's List interviews. After discussing it with the presenters and nominees, I picked a spot to stand out of their field of view with a good line of sight to the judges and tried to vanish into the wall. I can give useful feedback to the Dean's List Nominees based on my experience conducting job interviews. For the Chairman's team, basically all I can do is take notes on the presentation, record what the judges asked and how the presenters responded, and attempt to face-read the judges. I can't give any objective feedback on the quality of our overall Chairman's submission.

Depending on the judges and the event, the feedback form was of highly variable utility, but it was at least some outside feedback. I'd be happy to settle for one thing they liked and one area to improve, but we really need something.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I can confirm that the HOF teams as a whole agree that providing feedback on the Chairman's Award presentations is important.

One of the things we are proposing to Frank, is how we can be involved in the feedback process. With further discussions to take place and hopefully soon, I hope that the feedback can be implemented once again, whether it stays the same or in some revised format.

As a team that won the HOF in 2011, one of the most important critical factors that helped us improve year after year, was that feedback.

What I am very happy about despite no formal feedback process this past season, are teams taking the initiative to have peer to peer reviews with other teams that have won it in the past.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I tend to be data driven. If a team beats me, I'd like to know in what areas and by how much. It allows me to set goals for next time around. FLL does this with their review sheets with weighted scoring and comments. I believe FRC should move to the same model where different aspects of chairmans are weighted and scored. Numbers will never completely capture all the intangible portions of the award, but it still is priceless feedback for teams.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

Why not 'crowd source' the feedback? Since you can post everything, a post on CD asking for others (esp. HoF teams) to provide feedback, either publicly on CD or privately is entirely within reason. You might also get some judges to provide feedback in a less time pressured context. You should also be able to at least audio record the presentation with a smartphone to get the Q&A.

I clearly understand the desire for judge feedback. *I also understand how difficult it is for judges to provide quality feedback, especially if there are a lot of teams. In MN, you get 10 points toward State Tourney qualification just for applying for Chairmans. Not applying moves you down about 10 or so slots in the rankings. Winning Chairman's automatically qualifies you for the State tournament. This was done to incent teams to apply, but it does mean there are far more applicants now. Prior to the first state tourney in 2012, our team was at a Duluth regional where only 5 teams applied.

* I've never been an FRC judge but have been a debate judge and basically had about 5-7 minutes to write feedback for four kids other than what I wrote during the debate-- repeat four times during the day. I also had to take notes on the content of the debate arguments at the same time to assess which side won. It was very stressful. I can't imagine doing the similar for 20-30 or more teams applying at a 63 team regional. Some judges would find a reason not to return the next year.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

This was suggested at the Chairman's Chat in St. Louis, and the Judge Advisor there (I'm forgetting his name) said that the judges were specifically forbidden to do this. It seems like a great idea, but apparently it puts the judges in a pretty awkward position, so I'm not sure that it's the best course of action. That being said, I like your idea of crowd-sourcing the feedback - could be a great way to get feedback if some teams are willing to devote the time.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I personally liked the system, but I have nothing to compare it to. This is our team's first year submitting chairmans and the three of us that presented were all freshman.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I don't know the rules, but maybe it would not be an issue for a Chairman's judge to provide feedback to a team that they have not and will not be officially judging that year.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

Originally I was not a fan that teams would not be receiving feedback. For the most part our feedback over the years was telling us that we were just too young and our presentation was lacking in 2012 & 2013 and after focusing on the presentation along with another year of growth we finally won it at the district level in 2014. Just by having a mentor in the room we could have learned that our presentation wasn't the best and to a degree we knew it wasn't ready in 2012/2013.

After a season of really contemplating what the Chairmans award means and how a lot of teams go about receiving their feedback forms I partially agree with not receiving feedback at the district and regional level. The Chairmans award shouldn't be viewed as a metric sheet that if teams work to get high marks in certain areas or cater their off-season work towards areas they received low scores on. If a team wants to compete for the Chairmans award it needs to be a part of their team culture and they need to want to do their activities and outreach because they want to not because the sheet says they need to.

For teams presenting at the World Championship I would agree feedback should be provided. These are the role model teams in FIRST who do more than just focus on improving their team they are impacting their communities and the FIRST programs in them in powerful ways and continue to do so even after they become HOF teams.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

That's actually a really good idea. I might recommend to my team that this be implemented in the future!

To reflect on the changes, the time extension was helpful but not actually followed. At champs, the presenters walked in, introduced themselves, and stood there for a few moments before the judges said "We're starting the timers whenever you're ready". This was confusing because we were under the impression that the timers started when you walked in. After the presentation the judges wasted time asking if the members wanted to sit down, complimenting the presentation, and deciding who was to ask a question first, all while the timer was running. That being said, our team realized that next year we need to dedicate more of that time limit to questions, or else address major questions in the presentation instead of waiting for them to be asked. Additionally, the extra mentor was an opportunity our team found useless for the points mentioned earlier in this thread.


Like everyone else says, the lack of feedback was upsetting. No further lamentations are needed.


As a side note, the presentations rooms were very close together and it's understandable that teams wanted to cheer on their presenters afterwards. But as a presenter, it's hard enough to maintain focus with the pits below the rooms, let alone loud cheering just outside the door for sustained periods of time. Just please be considerate of other teams presenting.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

As Frank mentioned in today's blog, the Chairman's Feedback is returning for the 2016 season! You can take a look at the form here: http://www.firstinspires.org/sites/d...edbackForm.pdf

Please note that feedback is being given out on an "opt-in" basis. If your team wants feedback on your submission you need to fill in your team information on the form and provide it to the judges. Hopefully teams will take advantage of this new opportunity and use it to iterate and improve their submissions.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

This is great news! Thanks to all that made this happen. (HoF teams, I think?)

Does anyone know why this is a thing, rather than giving feedback to all teams by default? Also, will teams submitting at the FIRST Championship have the option of receiving feedback there?

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

from Frank's Blog: thank you, Karthik, for helping to bring this back. It is a much needed tool for teams.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

I really like the overhaul of the feedback form. I don't think that's a really edgy opinion, but I never thought the assigning of a 1-10 value for various aspects of the submission was anything but largely arbitrary.

I think the last question might have more potential if it was split into 2 or 3 different questions, but that move could as easily be unhelpful to all parties.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

If you've never tried writing feedback on things... it's REALLY freaking hard. Opt In means the judges only have to do it for folks who really want it and will use the info.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

Where were the Chairman's Essay and Executive Summary posted? I guess I missed that detail.

Re: Pilot Year Review: New Chairman's Award Feedback Structure
 

The 2015 Regional and District Championship Chairman's Award winner essays/videos/presentations can be found here: http://www.firstinspires.org/node/4881