[FRC Blog] Chairman’s Award Feedback

#1

Posted on the FRC Blog, 1/22/15: http://www.usfirst.org/roboticsprograms/frc/blog-chairmans-award-feedback

Chairman’s Award Feedback

**Blog Date: **Thursday, January 22, 2015 - 09:20

In this blog I announced changes to the feedback teams submitting for the Chairman’s Award would receive this season. We have received some feedback about these changes, from individual teams as well as our Hall of Fame teams, expressing concern about the lack of written feedback for teams this year. It’s fair to raise questions about this. Eliminating the written feedback was a judgment call we made based on several factors. We believe, on balance, we have made the right decision, so we will be continuing with the current plan for 2015. However, we are considering this change to be a pilot for this season, and we will be surveying the teams afterwards, on this specific change and others we made related to the Chairman’s Award, to see if additional changes need to be made in 2016.

Our Chief Judge Advisors, Cindy Stong and Allen Bancroft, offered to write a note for this blog expressing their thoughts about this. I accepted their offer. Please see their note below:

The judges want to give some insight to our decision to do a pilot to eliminate Chairman Award written feedback for the 2015 season…

We’ve seen some phenomenal growth in the quantity and quality of team applications for Regional / District Chairman Awards and Championship Chairman Awards (CA). We’ve allowed multiple opportunities to interview for CA which greatly benefits teams, and historically Judges have provided written feedback to teams regarding their interview. However, we have seen many issues with these feedback forms, including students who are very upset by the feedback given. In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…

We believe that our CA judges’ time is much better spent listening to the CA presentation and asking clarifying questions of students, and we believe that there are so many great resources available for teams seeking to improve their chances to earn the Chairman’s Award.

Here are some great changes to the 2015 Chairman’s Award judging process that teams should find beneficial:

Your team will get an extra 2 minutes in the CA room with the judges for extra presentation.

*(This allows you more wonderful opportunity to share the great things you are doing towards spreading the value of FIRST)
*
We will continue to allow a mentor into the CA interview room for silent observation.

(This is huge – what better feedback than having your mentor, who knows you so well, listen not only to what your team is saying but especially to the questions the judges are asking. These questions are the best feedback you will ever get. Your mentors know your team and are able to give you objective and accurate feedback. Please encourage your mentor to be in the room!)

We will review and add to our existing CA resources, such as update the CA resources webpage and hold live calls and webinars with CA Judges.

(We will continue to offer live calls where you can hear directly from Championship CA judges who have years of CA judging experience. These judges have seen so many great teams and they have a lot of insight to offer.)

Teams learn best from each other.

(Your team can learn so much from the Hall of Fame teams, from other ‘team’ resources [for example I watched the two hour Simbotics video on Chairman’s and I learned so much!], and from each other. And we are creating a way for teams to share official CA videos and practice interview videos with each other.)

Good luck this year to all of the many teams doing FIRST! And take advantage of the increased opportunity to get real-time feedback from your judges and feedback from each other.

Sincerely,

Your FRC Chief Judge Advisors,

Cynthia Stong and Allen Bancroft

Please be sure to carefully review Section 6.4 of the FRC Admin Manual for all the rules around the Chairman’s Award. There have been some important changes for 2015!

I want to thank Cindy and Allen (FRC Volunteers of the Year winners at the 2014 *FIRST *Championship, by the way) for their continued dedication to the program and tremendous work in guiding our judging process through some significant changes over the last few years. We couldn’t have done it without them!

Frank

#2

Ok,

I am a bit interested in knowing how many teams have mentors in the presentation rooms.

-Daniel

#3

It’s perfectly legal per the manual-however, “Not more than three (3) team members, of which one (1) may be an adult mentor, are allowed to attend the interview” so essentially you’d then have to cut down to two presenters. Unless you require translation services, then you can inconsequentially have a mentor there. I’ve never heard of a team doing it in the first case, and to me, it sounds like a huge disadvantage to do that.

Still incredibly against the change of eliminating feedback.

#4

Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

#5

This doesn’t make me any happier about getting rid of feedback. A mentor being in the room is a huge disadvantage, if you have to take away a presenter spot from a student. I’ve never heard of a chairman’s award winning team that had a mentor in the room, has anyone else?

I’m glad they are trying to find ways for teams to continue to improve but these are all general things and not specific feedback. It takes someone going over your entire submission to really give you feedback on both what they liked and what they thought you could improve on. Looking at other winning teams’ presentations and essays is a good idea but we also don’t want every presentation and every team to look the same. To me that’s what we are encouraging by not giving individual feedback but instead saying here look at these teams that did it right. This won’t encourage innovation since you have no way to know if something worked unless you happen to win and even then you don’t know if your innovative presentation style or essay writing was noticed by the judges.

#6

The blog first states,

“In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…”

They then go on to only list one clear way to get any sort of feedback, and that’s by having a mentor observe the presentation. This is something not many teams will probably do, considering the mentor would have to take the place of one of the three presenters. Even then, the mentor’s feedback will most likely not help the team majorly improve their chances of winning Chairman’s in the future, since they [the mentor] don’t know what the judges are specifically looking for in teams.

Since they only gave one, poorly elaborated upon reason as to why they removed the feedback forms, does anyone have any potential other reasons? This just seems like a largely unfavorable change.

#7

They did it largely because the judges were being over burdened. There were events that had Chairman’s Award judging happening on Saturday morning because they couldn’t fit all the presentations in on Friday. I completely understand the problem facing the judges but to me this solution will negatively affect teams. Another reason they did it was because the feedback form at some events wasn’t filled out well and only the check boxes were used by the judges. At other events we received typed feedback that was well thought out and helped develop our students and our team.

The feedback form had problems but eliminating it isn’t the answer. If you want feedback back please contact FIRST and let them know, talking about it on CD only does so much. Tell Frank when you see him or the judges or anyone else involved in making this decision, not everyone reads CD and knows people are upset by this.

#8

Tell Frank, judges have very little say.

#9

While that is certainly something that should have been addressed, as you said, eliminating the form altogether wasn’t the entirely appropriate answer. No feedback at all is probably less constructive than just some quick feedback from the judges about each presentation. But even then, taking 30 seconds to jot some things down for each team may stack up to be just as burdening as the forms were beforehand.

#10

I’ve had a lot of time to think about this decision to eliminate the Chairman’s Feedback Form and I have to say I’m no more pleased with it than I was when it first came out. The feedback form was the only method for teams to get any sort of official and concrete information on the strength and weaknesses of a submission. I know for 1114, the feedback forms we received over the years helped lead us to new levels of accomplishment and were a critical to our path to the Hall of Fame. Not only did they help us improve our submission, but they also helped lead our program in new directions which allowed us to come closer to our goal of achieving culture change. Judges gave us concrete examples of things we could do to help strengthen our program and we took that feedback and ran with it. It saddens me that teams are now going to miss this opportunity for feedback and growth, or at least have it limited in some fashion or another. The feedback form provided a direct path of communication between our team and the judges that has now been eliminated. In 2012 our feedback form told us that we “need[ed] to show more examples of diversity”. This was an easy change for us to make to our submission, but without the feedback form we never would have known this was something we were lacking or that the judges were stressing. There was no other way of knowing this.

At its core FIRST is an educational program for students. We want to see the students learn as much as possible through their participation in the program. In the classroom the feedback loop between teachers and students via the grading process is integral to the learning experience of the students. Especially in courses that focus on writing and presenting skills, students thrive on feedback from their teachers which allow them to develop and improve their skills over time. The Chairman’s process should be no different. We’ve all witnessed numerous students evolve by being part of a Chairman’s submission. They’ve become talented writers and presenters, and used these skills to earn scholarships and job offers at the top universities and companies across the world. But by eliminating the feedback form, we’ll be depriving these students of an incomparable experience. Where else other than FIRST do students have the opportunity to get feedback on their presenting skills from executives from Fortune 500 companies and leaders in our community? The Chairman’s feedback form represents a unique facet of the FIRST® program that we should not let go of.

Consider the analogy to the classroom; imagine that students work for an entire semester on major paper and presentation. After everything has been submitted and evaluated, the teacher steps up to the front of the classroom and announces the name of the student with the best overall submission. No one else is given any grades, feedback, or comments on their work. This would leave the other students feeling slightly dumbfounded and definitely with no sense of how their work was received or what they should improve in the future.

While 2 extra minutes of presenting time will be a welcome relief for many teams, this comes at the expense of 2 minutes of question time with the judges. This means the actual interaction with the Judges, a real avenue for feedback, will be cut down to 3 minutes for teams who choose this approach. This provides even less feedback for teams.

Leaving aside that the current rules only allow you have a mentor in the room at the expense of a presenting student, I don’t think this is a particularly useful avenue for feedback. By the time of the actual presentation, the mentor will have seen the presentation over a hundred times. They have had plenty of opportunity to offer feedback. Yes, this gives them a chance to give the presenters feedback on how they answered the questions, but that’s all. The mentors don’t know and understand the Chairman’s criteria they way the judges do, their feedback is just conjecture based on their best understanding of the award. For younger teams who are new to this process, that level of understanding can be tenuous at best. Regardless, this only addresses the presentation. Where’s the feedback on the essay coming from? Yes teams can have the essay read by teachers and mentors, but that’s still not going to help them understand what the judges are looking for.

I fail to see how removing the feedback form and allowing teams to present for longer at the expense of the question period has increased the opportunity for real-time feedback from the judges. In fact, I think it’s pretty clear that there will be substantially less room for feedback this season.

Giving proper feedback takes time. I fully understand that our judges are volunteers who are extremely busy. With some events having upwards of 40 Chairman’s Submissions, giving feedback to all teams based on their entire submission must be an incredibly time consuming and difficult process. It is definitely asking a lot of the judges to have to complete these forms. At the same time the Chairman’s Award is FIRST’s most prestigious and important award. If we’re going to hold this award as high as we do, and expect teams to hundreds if not thousands of people-hours into their efforts and submissions, I don’t think it’s too much to ask that they still receive a short feedback form that summarizes their strengths and weaknesses. If the check box rubric is too rigid and quantitative for tastes of some, then do away with it. Go with free form text boxes for strengths and weaknesses. It’s still essential that teams get back some comments from the judges to help them know how to improve, not just when it comes to Chairman’s submissions, but also in their own personal development of their writing and presenting skills.

If the workload is the issue, perhaps we can remove the expectation that feedback forms be handed back at the end of the event. Perhaps they can be emailed to the teams a few days after the event. Maybe we could have one person just do feedback on the essay (which could be done long before the event and only done once a season), and one person do feedback on the presentation. Thereby splitting and lessening the load on the feedback writers. I’m sure as a community we could come up with numerous ideas to streamline this process.

#11

Actually, helping to brainstorm for ideas to assist FIRST in this burdensome task is a great idea. I like the idea of separate essay & presentation feedback, for instance.

One thing that crossed my mind - a “court reporter” type position that can take the judges’ quick notes on a team’s essay and presentation and type it up on behalf of the judges. This could be available after the event for teams that want to retrieve it.

I see it working this way: The reporter is not a judge but merely taking notes. After a team’s presentation, the judges can discuss quickly among each other and communicate a few small things about the team’s presentation, and pass that along to the reporter (or the reporter can be present while they are talking). The reporter types this up in a standard feedback form, and when finished, these forms are provided to Pit Admin in an envelope for teams to pick up.

There are probably a few issues with this idea. Privacy is one that immediately comes to mind. But perhaps this volunteer doesn’t necessarily even need to be in the room during the presentations. If the judges keep track of the order in which teams present, the reporter can assign a letter or number on the feedback pages, and the Judge Advisor can stuff the appropriate team’s feedback into their already-numbered envelope. The reporter never sees more than an A-B-C letter.

Another issue is that this is a somewhat trusted and difficult position to fill. You need someone that is professionally literate and able to communicate the judges’ feedback - but maybe this would be a great entry position for a corporate volunteer with little FIRST experience. Someone that types quickly and would be able to meet that strict deadline is also helpful. :slight_smile:

This feedback might not be more than a couple sentences, but it is something.

Anyway. A quick idea, perhaps not the best, but the first thing that came to mind. Feel free to expand or poke holes through it. What else can we come up with, CD?

#12

I’ve been meaning to post this paper for about a week or so, but it’s a self eval form a few of us put together.

http://www.chiefdelphi.com/media/papers/3095

#13

I wanted to just let this go, but since the soapbox seal has been justly broken and I still find the decision… perplexing… I’ll fire up my old soapbox for the first time in a while.

In the sixth grade, I had a science teacher that had a lesson the first day to help refresh the idea of the scientific method in our young minds. The tool to help us was a menacing black wooden crate casting a shadow from the front table onto the floor below, affectionately called “Die Wunderbox”. The question: what would happen if he poured clear water through a funnel in the box? We took time observing the box, and due to its large nature, its rusting MasterLock keeping its contents hidden from us, and the teacher’s general wackiness, we hypothesized that the squirrely tube at the base of the box would output something different into a glass beaker on the floor, transforming the clear water he would pour into a funnel on the top. Our hypothesis was supported by the test. Clear water went in, and a lot of dyed water came out. Through this single input into the black box, we received a different output. How did the box change the input on the way to its output? He promised he would show us on the last day of eighth grade.

Almost 3 years later, June rolls around and the whole 8th grade class of around 100 students crowds Mr. Longworth’s room to see him reveal to us the inner workings of the mystery black box. For years we wanted to see a peek behind the curtain, a hint of what it could be. He pulls the key out of his pocket, jiggles it into the lock, and turns all the tumblers. The air is still in the room as the door swings open. Then we finally all see the secret…

It was a bright haired, wrinkly plastic troll doll, grasping a sign reading “Nice try, Suckers!” After all the anticipation, after youthful curiosity had reached its zenith, we literally got trolled. He took out the mechanism that gave us our output, for no reason other than a couple laughs at anguished students at the height of puberty.

FRC judging is kind of like the menacing, splintery black box. Teams spend all year preparing for a couple sessions where they can provide input to the judges. For all judged awards except one, the inputs of time and effort into executing robot design and growing the team’s outreach and reaching organizational milestones were funneled into the black box all at once, and out came all the awards winners. All except for one. The “black box” for the Chairman’s Award was never fully clear, but it was a far cry from the black box. Through its fuzzy, translucent shell one could reasonably see shadows of how their inputs of the written submission, presentation, and Q&A would generate a slightly clearer output than one you would get from the splintery, opaque box used for all the other awards.

422 was able to take the output of feedback from the judges over a 3 year period and turn a total dud of a team into a good team with a total dud of a submission, into a team that took home the award for the first time in its 15 years. Still, we kept trying to pry open the box; cut the lock, break the glass, just see what was really happening inside the Wonder Box of Chairman’s decision making. Instead, we come back later to see the box ajar, ready to have its lid peeled open and reveal…

a troll… a bright haired, wrinkly plastic troll doll, grasping a sign reading “Nice try, Suckers!”

1 Like
#14

I like this form. It’s more specific than the form that FIRST used up until this year, and it ditches the range of 10 checkboxes for each criterion for three checkboxes and a place for comments. In my experience, people pretty much took a 9 the same as a 10 on those forms, so this provides an opportunity for the judges to be more specific. I also like the spot for presentation-specific feedback, since that’s the one aspect of your submission that you can change in between events.

#15

From Franks section of the blog post…“We have received some feedback about these changes from individual teams, as well as our Hall of Fame teams, expressing concern about the lack of written feedback for teams this year”

From the judges note in the blog post…"Teams learn best from each other.

(Your team can learn so much from the Hall of Fame teams)"

Maybe we should listen to the Hall of Fame teams then. Like Karthik and Amanda showed, there are solutions. Give the HoF teams a chance to have some input on the matter.

#16

That doesn’t seem like a very positive change, maybe the Admin Manual is in error, keep an eye out for a possible update.

#17

Agreed, I’m assuming this is just an oversight based on what they said in the blog.

#18

I definitely think the option of a silent mentor attending should not eliminate a presenter. It kinda sorta made sense before when it was introduced, but removing the feedback form means we have to make other changes as well, and this would be one I would expect. I would also like to see this optional mentor position be opened up to more than just a team mentor. Allow any option silent observer.

This would allow teams (hall of fame, or teams that seem to always win a regional chairman’s award, etc) to offer up their services to others at their regional to provide feedback. I think even more important than feedback from the judges is feedback from the teams who have already achieved what your striving for. They know how much work it is, they know what they did and probably have a good idea of what you do (how many judges follow other team’s Twitter and Facebook accounts to keep up to date on local activities?), and they probably know and understand any limitations your team is dealing with. They can tell you what areas you are weak in, and highlight items that you may have done that didn’t come across in the presentation. Unlike judges, they can be around to help you team practice and improve in the off season, or able to sit in before the event next year to provide extra feedback. And on the plus side, they haven’t already seen the presentation a half hundred times!

#19

That is an excellent idea. Allowing teams to be critiqued by their (more accomplished) peers.

#20

I too find this removal of feedback disturbing… and their justifications sadly lacking. I also concur that the ideas mentioned so far regarding (1) feedback on the written submission which could be done once before competition as well as the idea of (2) assigning someone to serve as a “court reporter” to type up individual team feedback. These are really good and helpful ideas and if we continue working together on this topic I suspect we can come up with more.

I do have one one question that I have not heard yet…
Having judged at many FLL tournaments both local and state I am intimately familiar with their scoring rubrics and the judging process used at that level. And through that process I have learned that the perception of individuals viewing the same materials and hearing the same facts can vary widely based on their personal experience and area of expertise. Don’t the FRC Chairman’s judges use some sort of objective scoring system (eg. a rubric of sorts) to narrow down the field for discussion during their deliberation process? It seems to me that providing access to that score sheet at least provides some specific feedback to the team on how well their message was communicated and therefore if their presentation/submission was received as intended. Providing at least this portion of feedback seems like it would require almost no effort since these scores are being accumulated anyway. Maybe if we could combine this “low cost” rubric-like numeric scoring feedback with the somewhat off-loaded “court reporter” free-form subjective feedback, it seems like we would be doing everyone in the community a service.

Providing no feedback what so ever creates a process that lacks transparency. This can breed the appearance of favoritism and unfairness because there is no evidence refute it. This is not the culture change we are all striving for.