Pilot Year Review: New Chairman's Award Feedback Structure

[Hot_Copper_Frog]

Now that we've had a full season of Chairman's submissions, and experienced the full changes of the pilot feedback system, what are your thoughts? The new setup now includes:

• Regional and District Championship winners have their essay/executive summary and chairman’s video posted online. It was strongly encouraged that winning teams submit a practice presentation video as well, though not every team took part in that.
• The team is allowed an extra two minutes to present, if they so choose (5-7 minutes), though the 10 minuted total limit stayed the same.
• A mentor was allowed to sit in on the presentation session without counting against the 3-presenter limit. The mentor was theoretically allowed to film the session, but from what I understand that wasn’t true at all events.
• No formal feedback from the judges at any event.

So, what are your thoughts? Do you like it? Do you hate it? Was it as bad as you thought it was going to be? Were there unexpected benefits? And what direction do you think FIRST should consider moving next year with this?

Previous discussion about the decision when it was announced here: http://www.chiefdelphi.com/forums/sh...d.php?t=133343

FRC Blog post announcing the change here: http://www.usfirst.org/roboticsprogr...award-feedback

[Jay O'Donnell]

Originally Posted by Hot_Copper_Frog

Now that we've had a full season of Chairman's submissions, and experienced the full changes of the pilot feedback system, what are your thoughts? The new setup now includes:

• Regional and District Championship winners have their essay/executive summary and chairman’s video posted online. It was strongly encouraged that winning teams submit a practice presentation video as well, though not every team took part in that.
• The team is allowed an extra two minutes to present, if they so choose (5-7 minutes), though the 10 minuted total limit stayed the same.
• A mentor was allowed to sit in on the presentation session without counting against the 3-presenter limit. The mentor was theoretically allowed to film the session, but from what I understand that wasn’t true at all events.
• No formal feedback from the judges at any event.

So, what are your thoughts? Do you like it? Do you hate it? Was it as bad as you thought it was going to be? Were there unexpected benefits? And what direction do you think FIRST should consider moving next year with this?

Previous discussion about the decision when it was announced here: http://www.chiefdelphi.com/forums/sh...d.php?t=133343

FRC Blog post announcing the change here: http://www.usfirst.org/roboticsprogr...award-feedback

I very much disliked not getting a feedback form. It's the only real way to know how to improve your team. Having a mentor in the room only helps you get better with your presentation skills, it doesn't help the team as a whole move forward. How are we supposed to continue moving forward as a community to change the world if we don't know how to improve ourselves?

All of the other changes were excellent. No problems with any of those.

[Steven Donow]

Regional and District Championship winners have their essay/executive summary and chairman’s video posted online. It was strongly encouraged that winning teams submit a practice presentation video as well, though not every team took part in that.

This was nice, and it'll definitely help teams in the future.

The team is allowed an extra two minutes to present, if they so choose (5-7 minutes), though the 10 minuted total limit stayed the same.

This is perfectly acceptable and just plain makes sense to not have 5 minutes be a super-hard limit.

A mentor was allowed to sit in on the presentation session without counting against the 3-presenter limit. The mentor was theoretically allowed to film the session, but from what I understand that wasn’t true at all events.

This is a good change, although I heard absolutely nothing all year from any source about the mentor being allowed to film the session.

No formal feedback from the judges at any event.

I still just don't understand this. There's absolutely no legitimate way for a team to gauge how a judge ACTUALLY felt the team did on not just their presentation, but their essay as well. Getting to see who 'beat' (in quotes because FIRST is trying to push not viewing Chairman's as a competition...) you is not an acceptable form of feedback at all. It doesn't tell you what your team's weakness was; and gives nothing for a team to build on in order to improve at presenting their material.

[BigJ]

After giving it a go this season and discussion with the team we preferred the feedback forms, mainly because we didn't use the mentor-in-room for a number of reasons.

We realized before the presentation that having a mentor observe would only add an "harder" step of guessing:

2014: Present -> Feedback Form -> Attempt to discern what more the judges wanted -> Improve presentation for next regional/year
2015: Present -> Mentor listens -> Mentor attempts to discern how presentation/answers could improve by trying to "read judges" -> Improve presentation for next regional/year

Most of the improvements a mentor would see outside of "reading the judges" would be presentation flow, answers to questions, etc that we had been practicing and working on for weeks already (in our opinions). There wasn't a lot there that would become "newly apparent" to us. I can't imagine a situation where the mentor would be able to glean more than the students in how the presentation/answers could be improved, but then again we didn't send a mentor in.

Between the 2 reasons above, concerns about the viewing mentor being able to keep a straight/neutral face through the presentation (if a line were missed or question were flubbed) as to not affect it, and other assorted issues, we decided to let the presenters go in the room and do their thing without the additional pressure.

I think they could do away with the giant however-many category rubric (those never helped us anyway) and just have a few short written feedback questions like the ones in previous years..

Flexible presentation/question time was good. We aimed to have max time available for Q&A, but not having to hit 5:00 on the relative dot was much better for us this year.

[Dan Petrovic]

Us at Chop Shop dislike the lack of feedback. An outsider's perspective on what is working and what isn't is great to have.

More often than not, it really helps knowing where we can focus our efforts in the off-season or if we're just totally missing the mark all together.

[Riverdance]

Most people that I've spoken to have pointed out the lack of feedback as a negative, and I have to agree, for many reasons. That being said, I attended the Chairman's Chat presentation at Championship and the same issue was brought up. Karthik responded that the Hall of Fame teams spoke to FIRST about the issue when the decision was first made and they were surprised that it was being considered an issue, as only one team besides the aforementioned Hall of Fame teams had spoken out by contacting FIRST itself.

When we take issue with a policy that FIRST implements, it is our duty as teams to speak out. Clearly, we all thought that everyone else was going to send an email about it and didn't bother doing it ourselves. The clear solution here is to voice our passion to FIRST. They aren't going to fix it if they don't know that it's a problem.

[Hot_Copper_Frog]

Originally Posted by Riverdance

Most people that I've spoken to have pointed out the lack of feedback as a negative, and I have to agree, for many reasons. That being said, I attended the Chairman's Chat presentation at Championship and the same issue was brought up. Karthik responded that the Hall of Fame teams spoke to FIRST about the issue when the decision was first made and they were surprised that it was being considered an issue, as only one team besides the aforementioned Hall of Fame teams had spoken out by contacting FIRST itself.

When we take issue with a policy that FIRST implements, it is our duty as teams to speak out. Clearly, we all thought that everyone else was going to send an email about it and didn't bother doing it ourselves. The clear solution here is to voice our passion to FIRST. They aren't going to fix it if they don't know that it's a problem.

That's interesting. I know on my part, I wasn't aware of the changes until they were already implemented as policy. Looks like I need to keep a closer eye on what's happening at an organizational level.

In that vein, it may not be a bad idea to submit an organized grouping of feedback from teams regarding the policy change after experiencing it all the way through the season.

[Andrew Schreiber]

Originally Posted by Riverdance

Most people that I've spoken to have pointed out the lack of feedback as a negative, and I have to agree, for many reasons. That being said, I attended the Chairman's Chat presentation at Championship and the same issue was brought up. Karthik responded that the Hall of Fame teams spoke to FIRST about the issue when the decision was first made and they were surprised that it was being considered an issue, as only one team besides the aforementioned Hall of Fame teams had spoken out by contacting FIRST itself.

When we take issue with a policy that FIRST implements, it is our duty as teams to speak out. Clearly, we all thought that everyone else was going to send an email about it and didn't bother doing it ourselves. The clear solution here is to voice our passion to FIRST. They aren't going to fix it if they don't know that it's a problem.

I question this. I know for a fact I contacted FIRST (in my capacities as a Judge Advisor, Planning Committee member, and mentor) and I find it very hard to believe we were the only ones.

It's also completely at odds with the feedback I got from Frank in which he was well aware that the lack of feedback was a problem (I received this feedback two distinct times, during build a week after the announcement was made and during a feedback session at Championships)

In fact, I know that FIRST is being pushed to provide even more feedback. DL and Entrepreneurship were the top of my list but I'd like to see some form of feedback about the interviews conducted by judges. HQ has been made aware of this but there are a ton of logistical issues with implementing more feedback to teams.

I'll tell you the same feedback I've said all along, it's a step backwards. I understand why the step was made but I hope that we can come up with a solution to those problems and give teams back their feedback.

[Taylor]

The current method is ridiculous. A team mentor is going to focus on how the presentation was presented, not the content of said presentation.
If I had it to do over again, I'd have a mentor from a friendly but separate team sit in our Chairman's presentation, and offer to do the same for their team.

[BlueLipstick]

Originally Posted by Taylor

If I had it to do over again, I'd have a mentor from a friendly but separate team sit in our Chairman's presentation, and offer to do the same for their team.

That's actually a really good idea. I might recommend to my team that this be implemented in the future!

To reflect on the changes, the time extension was helpful but not actually followed. At champs, the presenters walked in, introduced themselves, and stood there for a few moments before the judges said "We're starting the timers whenever you're ready". This was confusing because we were under the impression that the timers started when you walked in. After the presentation the judges wasted time asking if the members wanted to sit down, complimenting the presentation, and deciding who was to ask a question first, all while the timer was running. That being said, our team realized that next year we need to dedicate more of that time limit to questions, or else address major questions in the presentation instead of waiting for them to be asked. Additionally, the extra mentor was an opportunity our team found useless for the points mentioned earlier in this thread.


Like everyone else says, the lack of feedback was upsetting. No further lamentations are needed.


As a side note, the presentations rooms were very close together and it's understandable that teams wanted to cheer on their presenters afterwards. But as a presenter, it's hard enough to maintain focus with the pits below the rooms, let alone loud cheering just outside the door for sustained periods of time. Just please be considerate of other teams presenting.

[Karthik]

As Frank mentioned in today's blog, the Chairman's Feedback is returning for the 2016 season! You can take a look at the form here: http://www.firstinspires.org/sites/d...edbackForm.pdf

Please note that feedback is being given out on an "opt-in" basis. If your team wants feedback on your submission you need to fill in your team information on the form and provide it to the judges. Hopefully teams will take advantage of this new opportunity and use it to iterate and improve their submissions.

[Jacob Bendicksen]

This is great news! Thanks to all that made this happen. (HoF teams, I think?)

Originally Posted by Karthik

Please note that feedback is being given out on an "opt-in" basis. If your team wants feedback on your submission you need to fill in your team information on the form and provide it to the judges.

Does anyone know why this is a thing, rather than giving feedback to all teams by default? Also, will teams submitting at the FIRST Championship have the option of receiving feedback there?

[wilsonmw04]

Originally Posted by Jacob Bendicksen

Does anyone know why this is a thing, rather than giving feedback to all teams by default? Also, will teams submitting at the FIRST Championship have the option of receiving feedback there?

from Frank's Blog:

In prior years, when written feedback was completed on every team, Judges have reported several instances of teams not picking up their feedback forms at the end of events. While some of these instances may have been communications problems, there was a sense that some teams just weren’t interested in feedback,

thank you, Karthik, for helping to bring this back. It is a much needed tool for teams.

[PayneTrain]

I really like the overhaul of the feedback form. I don't think that's a really edgy opinion, but I never thought the assigning of a 1-10 value for various aspects of the submission was anything but largely arbitrary.

I think the last question might have more potential if it was split into 2 or 3 different questions, but that move could as easily be unhelpful to all parties.

[Andrew Schreiber]

Originally Posted by Jacob Bendicksen

This is great news! Thanks to all that made this happen. (HoF teams, I think?)



Does anyone know why this is a thing, rather than giving feedback to all teams by default? Also, will teams submitting at the FIRST Championship have the option of receiving feedback there?

If you've never tried writing feedback on things... it's REALLY freaking hard. Opt In means the judges only have to do it for folks who really want it and will use the info.