[FRC Blog] Chairman’s Award Feedback

[Hallry]

Posted on the FRC Blog, 1/22/15: http://www.usfirst.org/roboticsprogr...award-feedback

Chairman’s Award Feedback

Blog Date: Thursday, January 22, 2015 - 09:20

In this blog I announced changes to the feedback teams submitting for the Chairman’s Award would receive this season. We have received some feedback about these changes, from individual teams as well as our Hall of Fame teams, expressing concern about the lack of written feedback for teams this year. It’s fair to raise questions about this. Eliminating the written feedback was a judgment call we made based on several factors. We believe, on balance, we have made the right decision, so we will be continuing with the current plan for 2015. However, we are considering this change to be a pilot for this season, and we will be surveying the teams afterwards, on this specific change and others we made related to the Chairman’s Award, to see if additional changes need to be made in 2016.

Our Chief Judge Advisors, Cindy Stong and Allen Bancroft, offered to write a note for this blog expressing their thoughts about this. I accepted their offer. Please see their note below:

The judges want to give some insight to our decision to do a pilot to eliminate Chairman Award written feedback for the 2015 season…

We've seen some phenomenal growth in the quantity and quality of team applications for Regional / District Chairman Awards and Championship Chairman Awards (CA). We've allowed multiple opportunities to interview for CA which greatly benefits teams, and historically Judges have provided written feedback to teams regarding their interview. However, we have seen many issues with these feedback forms, including students who are very upset by the feedback given. In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…

We believe that our CA judges' time is much better spent listening to the CA presentation and asking clarifying questions of students, and we believe that there are so many great resources available for teams seeking to improve their chances to earn the Chairman’s Award.

Here are some great changes to the 2015 Chairman’s Award judging process that teams should find beneficial:

Your team will get an extra 2 minutes in the CA room with the judges for extra presentation.

(This allows you more wonderful opportunity to share the great things you are doing towards spreading the value of FIRST)

We will continue to allow a mentor into the CA interview room for silent observation.

(This is huge – what better feedback than having your mentor, who knows you so well, listen not only to what your team is saying but especially to the questions the judges are asking. These questions are the best feedback you will ever get. Your mentors know your team and are able to give you objective and accurate feedback. Please encourage your mentor to be in the room!)

We will review and add to our existing CA resources, such as update the CA resources webpage and hold live calls and webinars with CA Judges.

(We will continue to offer live calls where you can hear directly from Championship CA judges who have years of CA judging experience. These judges have seen so many great teams and they have a lot of insight to offer.)

Teams learn best from each other.

(Your team can learn so much from the Hall of Fame teams, from other 'team' resources [for example I watched the two hour Simbotics video on Chairman's and I learned so much!], and from each other. And we are creating a way for teams to share official CA videos and practice interview videos with each other.)

Good luck this year to all of the many teams doing FIRST! And take advantage of the increased opportunity to get real-time feedback from your judges and feedback from each other.

Sincerely,

Your FRC Chief Judge Advisors,

Cynthia Stong and Allen Bancroft

Please be sure to carefully review Section 6.4 of the FRC Admin Manual for all the rules around the Chairman’s Award. There have been some important changes for 2015!

I want to thank Cindy and Allen (FRC Volunteers of the Year winners at the 2014 FIRST Championship, by the way) for their continued dedication to the program and tremendous work in guiding our judging process through some significant changes over the last few years. We couldn’t have done it without them!

Frank

[MysterE]

Ok,

I am a bit interested in knowing how many teams have mentors in the presentation rooms.

-Daniel

[Steven Donow]

Originally Posted by MysterE

Ok,

I am a bit interested in knowing how many teams have mentors in the presentation rooms.

-Daniel

It's perfectly legal per the manual-however, "Not more than three (3) team members, of which one (1) may be an adult mentor, are allowed to attend the interview" so essentially you'd then have to cut down to two presenters. Unless you require translation services, then you can inconsequentially have a mentor there. I've never heard of a team doing it in the first case, and to me, it sounds like a huge disadvantage to do that.

Still incredibly against the change of eliminating feedback.

[vhcook]

Originally Posted by Steven Donow

It's perfectly legal per the manual-however, "Not more than three (3) team members, of which one (1) may be an adult mentor, are allowed to attend the interview" so essentially you'd then have to cut down to two presenters. Unless you require translation services, then you can inconsequentially have a mentor there. I've never heard of a team doing it in the first case, and to me, it sounds like a huge disadvantage to do that.

Still incredibly against the change of eliminating feedback.

Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

[AllenGregoryIV]

This doesn't make me any happier about getting rid of feedback. A mentor being in the room is a huge disadvantage, if you have to take away a presenter spot from a student. I've never heard of a chairman's award winning team that had a mentor in the room, has anyone else?

I'm glad they are trying to find ways for teams to continue to improve but these are all general things and not specific feedback. It takes someone going over your entire submission to really give you feedback on both what they liked and what they thought you could improve on. Looking at other winning teams' presentations and essays is a good idea but we also don't want every presentation and every team to look the same. To me that's what we are encouraging by not giving individual feedback but instead saying here look at these teams that did it right. This won't encourage innovation since you have no way to know if something worked unless you happen to win and even then you don't know if your innovative presentation style or essay writing was noticed by the judges.

[CTbiker105]

The blog first states,

"In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…"

They then go on to only list one clear way to get any sort of feedback, and that's by having a mentor observe the presentation. This is something not many teams will probably do, considering the mentor would have to take the place of one of the three presenters. Even then, the mentor's feedback will most likely not help the team majorly improve their chances of winning Chairman's in the future, since they [the mentor] don't know what the judges are specifically looking for in teams.

Since they only gave one, poorly elaborated upon reason as to why they removed the feedback forms, does anyone have any potential other reasons? This just seems like a largely unfavorable change.

[AllenGregoryIV]

Originally Posted by CTbiker105

Since they only gave one, poorly elaborated upon reason as to why they removed the feedback forms, does anyone have any potential other reasons? This just seems like a largely unfavorable change.

They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday. I completely understand the problem facing the judges but to me this solution will negatively affect teams. Another reason they did it was because the feedback form at some events wasn't filled out well and only the check boxes were used by the judges. At other events we received typed feedback that was well thought out and helped develop our students and our team.

The feedback form had problems but eliminating it isn't the answer. If you want feedback back please contact FIRST and let them know, talking about it on CD only does so much. Tell Frank when you see him or the judges or anyone else involved in making this decision, not everyone reads CD and knows people are upset by this.

[Andrew Schreiber]

Originally Posted by AllenGregoryIV

They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday. I completely understand the problem facing the judges but to me this solution will negatively affect teams. Another reason they did it was because the feedback form at some events wasn't filled out well and only the check boxes were used by the judges. At other events we received typed feedback that was well thought out and helped develop our students and our team.

The feedback form had problems but eliminating it isn't the answer. If you want feedback back please contact FIRST and let them know, talking about it on CD only does so much. Tell Frank when you see him or the judges or anyone else involved in making this decision, not everyone reads CD and knows people are upset by this.

Tell Frank, judges have very little say.

[Karthik]

I've had a lot of time to think about this decision to eliminate the Chairman's Feedback Form and I have to say I'm no more pleased with it than I was when it first came out. The feedback form was the only method for teams to get any sort of official and concrete information on the strength and weaknesses of a submission. I know for 1114, the feedback forms we received over the years helped lead us to new levels of accomplishment and were a critical to our path to the Hall of Fame. Not only did they help us improve our submission, but they also helped lead our program in new directions which allowed us to come closer to our goal of achieving culture change. Judges gave us concrete examples of things we could do to help strengthen our program and we took that feedback and ran with it. It saddens me that teams are now going to miss this opportunity for feedback and growth, or at least have it limited in some fashion or another. The feedback form provided a direct path of communication between our team and the judges that has now been eliminated. In 2012 our feedback form told us that we "need[ed] to show more examples of diversity". This was an easy change for us to make to our submission, but without the feedback form we never would have known this was something we were lacking or that the judges were stressing. There was no other way of knowing this.

At its core FIRST is an educational program for students. We want to see the students learn as much as possible through their participation in the program. In the classroom the feedback loop between teachers and students via the grading process is integral to the learning experience of the students. Especially in courses that focus on writing and presenting skills, students thrive on feedback from their teachers which allow them to develop and improve their skills over time. The Chairman’s process should be no different. We’ve all witnessed numerous students evolve by being part of a Chairman’s submission. They’ve become talented writers and presenters, and used these skills to earn scholarships and job offers at the top universities and companies across the world. But by eliminating the feedback form, we’ll be depriving these students of an incomparable experience. Where else other than FIRST do students have the opportunity to get feedback on their presenting skills from executives from Fortune 500 companies and leaders in our community? The Chairman’s feedback form represents a unique facet of the FIRST® program that we should not let go of.

Consider the analogy to the classroom; imagine that students work for an entire semester on major paper and presentation. After everything has been submitted and evaluated, the teacher steps up to the front of the classroom and announces the name of the student with the best overall submission. No one else is given any grades, feedback, or comments on their work. This would leave the other students feeling slightly dumbfounded and definitely with no sense of how their work was received or what they should improve in the future.

Originally Posted by FRC Blog

In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…
...
Your team will get an extra 2 minutes in the CA room with the judges for extra presentation.

While 2 extra minutes of presenting time will be a welcome relief for many teams, this comes at the expense of 2 minutes of question time with the judges. This means the actual interaction with the Judges, a real avenue for feedback, will be cut down to 3 minutes for teams who choose this approach. This provides even less feedback for teams.

Originally Posted by FRC Blog

We will continue to allow a mentor into the CA interview room for silent observation.

(This is huge – what better feedback than having your mentor, who knows you so well, listen not only to what your team is saying but especially to the questions the judges are asking. These questions are the best feedback you will ever get. Your mentors know your team and are able to give you objective and accurate feedback. Please encourage your mentor to be in the room!)

Leaving aside that the current rules only allow you have a mentor in the room at the expense of a presenting student, I don't think this is a particularly useful avenue for feedback. By the time of the actual presentation, the mentor will have seen the presentation over a hundred times. They have had plenty of opportunity to offer feedback. Yes, this gives them a chance to give the presenters feedback on how they answered the questions, but that's all. The mentors don't know and understand the Chairman's criteria they way the judges do, their feedback is just conjecture based on their best understanding of the award. For younger teams who are new to this process, that level of understanding can be tenuous at best. Regardless, this only addresses the presentation. Where's the feedback on the essay coming from? Yes teams can have the essay read by teachers and mentors, but that's still not going to help them understand what the judges are looking for.

Originally Posted by FRC Blog

And take advantage of the increased opportunity to get real-time feedback from your judges and feedback from each other.

I fail to see how removing the feedback form and allowing teams to present for longer at the expense of the question period has increased the opportunity for real-time feedback from the judges. In fact, I think it's pretty clear that there will be substantially less room for feedback this season.

Giving proper feedback takes time. I fully understand that our judges are volunteers who are extremely busy. With some events having upwards of 40 Chairman's Submissions, giving feedback to all teams based on their entire submission must be an incredibly time consuming and difficult process. It is definitely asking a lot of the judges to have to complete these forms. At the same time the Chairman's Award is FIRST's most prestigious and important award. If we're going to hold this award as high as we do, and expect teams to hundreds if not thousands of people-hours into their efforts and submissions, I don't think it's too much to ask that they still receive a short feedback form that summarizes their strengths and weaknesses. If the check box rubric is too rigid and quantitative for tastes of some, then do away with it. Go with free form text boxes for strengths and weaknesses. It's still essential that teams get back some comments from the judges to help them know how to improve, not just when it comes to Chairman's submissions, but also in their own personal development of their writing and presenting skills.

If the workload is the issue, perhaps we can remove the expectation that feedback forms be handed back at the end of the event. Perhaps they can be emailed to the teams a few days after the event. Maybe we could have one person just do feedback on the essay (which could be done long before the event and only done once a season), and one person do feedback on the presentation. Thereby splitting and lessening the load on the feedback writers. I'm sure as a community we could come up with numerous ideas to streamline this process.

[Amanda Morrison]

Originally Posted by Karthik

If the workload is the issue, perhaps we can remove the expectation that feedback forms be handed back at the end of the event. Perhaps they can be emailed to the teams a few days after the event. Maybe we could have one person just do feedback on the essay (which could be done long before the event and only done once a season), and one person do feedback on the presentation. Thereby splitting and lessening the load on the feedback writers. I'm sure as a community we could come up with numerous ideas to streamline this process.

Actually, helping to brainstorm for ideas to assist FIRST in this burdensome task is a great idea. I like the idea of separate essay & presentation feedback, for instance.

One thing that crossed my mind - a "court reporter" type position that can take the judges' quick notes on a team's essay and presentation and type it up on behalf of the judges. This could be available after the event for teams that want to retrieve it.

I see it working this way: The reporter is not a judge but merely taking notes. After a team's presentation, the judges can discuss quickly among each other and communicate a few small things about the team's presentation, and pass that along to the reporter (or the reporter can be present while they are talking). The reporter types this up in a standard feedback form, and when finished, these forms are provided to Pit Admin in an envelope for teams to pick up.

There are probably a few issues with this idea. Privacy is one that immediately comes to mind. But perhaps this volunteer doesn't necessarily even need to be in the room during the presentations. If the judges keep track of the order in which teams present, the reporter can assign a letter or number on the feedback pages, and the Judge Advisor can stuff the appropriate team's feedback into their already-numbered envelope. The reporter never sees more than an A-B-C letter.

Another issue is that this is a somewhat trusted and difficult position to fill. You need someone that is professionally literate and able to communicate the judges' feedback - but maybe this would be a great entry position for a corporate volunteer with little FIRST experience. Someone that types quickly and would be able to meet that strict deadline is also helpful.

This feedback might not be more than a couple sentences, but it is something.

Anyway. A quick idea, perhaps not the best, but the first thing that came to mind. Feel free to expand or poke holes through it. What else can we come up with, CD?

[Andrew Schreiber]

Originally Posted by Amanda Morrison

What else can we come up with, CD?

I've been meaning to post this paper for about a week or so, but it's a self eval form a few of us put together.

http://www.chiefdelphi.com/media/papers/3095

[CTbiker105]

Originally Posted by AllenGregoryIV

They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday.

While that is certainly something that should have been addressed, as you said, eliminating the form altogether wasn't the entirely appropriate answer. No feedback at all is probably less constructive than just some quick feedback from the judges about each presentation. But even then, taking 30 seconds to jot some things down for each team may stack up to be just as burdening as the forms were beforehand.

[rsisk]

Originally Posted by vhcook

Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

That doesn't seem like a very positive change, maybe the Admin Manual is in error, keep an eye out for a possible update.

[Karthik]

Originally Posted by rsisk

That doesn't seem like a very positive change, maybe the Admin Manual is in error, keep an eye out for a possible update.

Agreed, I'm assuming this is just an oversight based on what they said in the blog.

[Jon Stratis]

I definitely think the option of a silent mentor attending should not eliminate a presenter. It kinda sorta made sense before when it was introduced, but removing the feedback form means we have to make other changes as well, and this would be one I would expect. I would also like to see this optional mentor position be opened up to more than just a team mentor. Allow any option silent observer.

This would allow teams (hall of fame, or teams that seem to always win a regional chairman's award, etc) to offer up their services to others at their regional to provide feedback. I think even more important than feedback from the judges is feedback from the teams who have already achieved what your striving for. They know how much work it is, they know what they did and probably have a good idea of what you do (how many judges follow other team's Twitter and Facebook accounts to keep up to date on local activities?), and they probably know and understand any limitations your team is dealing with. They can tell you what areas you are weak in, and highlight items that you may have done that didn't come across in the presentation. Unlike judges, they can be around to help you team practice and improve in the off season, or able to sit in before the event next year to provide extra feedback. And on the plus side, they haven't already seen the presentation a half hundred times!