Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   General Forum (http://www.chiefdelphi.com/forums/forumdisplay.php?f=16)
-   -   [FRC Blog] Chairman’s Award Feedback (http://www.chiefdelphi.com/forums/showthread.php?t=133343)

Hallry 22-01-2015 14:47

[FRC Blog] Chairman’s Award Feedback
 
Posted on the FRC Blog, 1/22/15: http://www.usfirst.org/roboticsprogr...award-feedback

Quote:

Chairman’s Award Feedback

Blog Date: Thursday, January 22, 2015 - 09:20

In this blog I announced changes to the feedback teams submitting for the Chairman’s Award would receive this season. We have received some feedback about these changes, from individual teams as well as our Hall of Fame teams, expressing concern about the lack of written feedback for teams this year. It’s fair to raise questions about this. Eliminating the written feedback was a judgment call we made based on several factors. We believe, on balance, we have made the right decision, so we will be continuing with the current plan for 2015. However, we are considering this change to be a pilot for this season, and we will be surveying the teams afterwards, on this specific change and others we made related to the Chairman’s Award, to see if additional changes need to be made in 2016.

Our Chief Judge Advisors, Cindy Stong and Allen Bancroft, offered to write a note for this blog expressing their thoughts about this. I accepted their offer. Please see their note below:
The judges want to give some insight to our decision to do a pilot to eliminate Chairman Award written feedback for the 2015 season…

We've seen some phenomenal growth in the quantity and quality of team applications for Regional / District Chairman Awards and Championship Chairman Awards (CA). We've allowed multiple opportunities to interview for CA which greatly benefits teams, and historically Judges have provided written feedback to teams regarding their interview. However, we have seen many issues with these feedback forms, including students who are very upset by the feedback given. In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…

We believe that our CA judges' time is much better spent listening to the CA presentation and asking clarifying questions of students, and we believe that there are so many great resources available for teams seeking to improve their chances to earn the Chairman’s Award.

Here are some great changes to the 2015 Chairman’s Award judging process that teams should find beneficial:

Your team will get an extra 2 minutes in the CA room with the judges for extra presentation.

(This allows you more wonderful opportunity to share the great things you are doing towards spreading the value of FIRST)

We will continue to allow a mentor into the CA interview room for silent observation.

(This is huge – what better feedback than having your mentor, who knows you so well, listen not only to what your team is saying but especially to the questions the judges are asking. These questions are the best feedback you will ever get. Your mentors know your team and are able to give you objective and accurate feedback. Please encourage your mentor to be in the room!)

We will review and add to our existing CA resources, such as update the CA resources webpage and hold live calls and webinars with CA Judges.

(We will continue to offer live calls where you can hear directly from Championship CA judges who have years of CA judging experience. These judges have seen so many great teams and they have a lot of insight to offer.)

Teams learn best from each other.

(Your team can learn so much from the Hall of Fame teams, from other 'team' resources [for example I watched the two hour Simbotics video on Chairman's and I learned so much!], and from each other. And we are creating a way for teams to share official CA videos and practice interview videos with each other.)

Good luck this year to all of the many teams doing FIRST! And take advantage of the increased opportunity to get real-time feedback from your judges and feedback from each other.

Sincerely,

Your FRC Chief Judge Advisors,

Cynthia Stong and Allen Bancroft
Please be sure to carefully review Section 6.4 of the FRC Admin Manual for all the rules around the Chairman’s Award. There have been some important changes for 2015!

I want to thank Cindy and Allen (FRC Volunteers of the Year winners at the 2014 FIRST Championship, by the way) for their continued dedication to the program and tremendous work in guiding our judging process through some significant changes over the last few years. We couldn’t have done it without them!

Frank

MysterE 22-01-2015 15:14

Re: [FRC Blog] Chairman’s Award Feedback
 
Ok,

I am a bit interested in knowing how many teams have mentors in the presentation rooms.

-Daniel

Steven Donow 22-01-2015 15:19

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by MysterE (Post 1432262)
Ok,

I am a bit interested in knowing how many teams have mentors in the presentation rooms.

-Daniel

It's perfectly legal per the manual-however, "Not more than three (3) team members, of which one (1) may be an adult mentor, are allowed to attend the interview" so essentially you'd then have to cut down to two presenters. Unless you require translation services, then you can inconsequentially have a mentor there. I've never heard of a team doing it in the first case, and to me, it sounds like a huge disadvantage to do that.

Still incredibly against the change of eliminating feedback.

vhcook 22-01-2015 15:41

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Steven Donow (Post 1432264)
It's perfectly legal per the manual-however, "Not more than three (3) team members, of which one (1) may be an adult mentor, are allowed to attend the interview" so essentially you'd then have to cut down to two presenters. Unless you require translation services, then you can inconsequentially have a mentor there. I've never heard of a team doing it in the first case, and to me, it sounds like a huge disadvantage to do that.

Still incredibly against the change of eliminating feedback.

Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

AllenGregoryIV 22-01-2015 15:54

Re: [FRC Blog] Chairman’s Award Feedback
 
This doesn't make me any happier about getting rid of feedback. A mentor being in the room is a huge disadvantage, if you have to take away a presenter spot from a student. I've never heard of a chairman's award winning team that had a mentor in the room, has anyone else?

I'm glad they are trying to find ways for teams to continue to improve but these are all general things and not specific feedback. It takes someone going over your entire submission to really give you feedback on both what they liked and what they thought you could improve on. Looking at other winning teams' presentations and essays is a good idea but we also don't want every presentation and every team to look the same. To me that's what we are encouraging by not giving individual feedback but instead saying here look at these teams that did it right. This won't encourage innovation since you have no way to know if something worked unless you happen to win and even then you don't know if your innovative presentation style or essay writing was noticed by the judges.

CTbiker105 22-01-2015 16:20

Re: [FRC Blog] Chairman’s Award Feedback
 
The blog first states,

"In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…"

They then go on to only list one clear way to get any sort of feedback, and that's by having a mentor observe the presentation. This is something not many teams will probably do, considering the mentor would have to take the place of one of the three presenters. Even then, the mentor's feedback will most likely not help the team majorly improve their chances of winning Chairman's in the future, since they [the mentor] don't know what the judges are specifically looking for in teams.

Since they only gave one, poorly elaborated upon reason as to why they removed the feedback forms, does anyone have any potential other reasons? This just seems like a largely unfavorable change.

AllenGregoryIV 22-01-2015 17:41

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by CTbiker105 (Post 1432295)
Since they only gave one, poorly elaborated upon reason as to why they removed the feedback forms, does anyone have any potential other reasons? This just seems like a largely unfavorable change.

They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday. I completely understand the problem facing the judges but to me this solution will negatively affect teams. Another reason they did it was because the feedback form at some events wasn't filled out well and only the check boxes were used by the judges. At other events we received typed feedback that was well thought out and helped develop our students and our team.

The feedback form had problems but eliminating it isn't the answer. If you want feedback back please contact FIRST and let them know, talking about it on CD only does so much. Tell Frank when you see him or the judges or anyone else involved in making this decision, not everyone reads CD and knows people are upset by this.

Andrew Schreiber 22-01-2015 17:47

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by AllenGregoryIV (Post 1432340)
They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday. I completely understand the problem facing the judges but to me this solution will negatively affect teams. Another reason they did it was because the feedback form at some events wasn't filled out well and only the check boxes were used by the judges. At other events we received typed feedback that was well thought out and helped develop our students and our team.

The feedback form had problems but eliminating it isn't the answer. If you want feedback back please contact FIRST and let them know, talking about it on CD only does so much. Tell Frank when you see him or the judges or anyone else involved in making this decision, not everyone reads CD and knows people are upset by this.

Tell Frank, judges have very little say.

CTbiker105 22-01-2015 19:03

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by AllenGregoryIV (Post 1432340)
They did it largely because the judges were being over burdened. There were events that had Chairman's Award judging happening on Saturday morning because they couldn't fit all the presentations in on Friday.

While that is certainly something that should have been addressed, as you said, eliminating the form altogether wasn't the entirely appropriate answer. No feedback at all is probably less constructive than just some quick feedback from the judges about each presentation. But even then, taking 30 seconds to jot some things down for each team may stack up to be just as burdening as the forms were beforehand.

Karthik 22-01-2015 19:08

Re: [FRC Blog] Chairman’s Award Feedback
 
I've had a lot of time to think about this decision to eliminate the Chairman's Feedback Form and I have to say I'm no more pleased with it than I was when it first came out. The feedback form was the only method for teams to get any sort of official and concrete information on the strength and weaknesses of a submission. I know for 1114, the feedback forms we received over the years helped lead us to new levels of accomplishment and were a critical to our path to the Hall of Fame. Not only did they help us improve our submission, but they also helped lead our program in new directions which allowed us to come closer to our goal of achieving culture change. Judges gave us concrete examples of things we could do to help strengthen our program and we took that feedback and ran with it. It saddens me that teams are now going to miss this opportunity for feedback and growth, or at least have it limited in some fashion or another. The feedback form provided a direct path of communication between our team and the judges that has now been eliminated. In 2012 our feedback form told us that we "need[ed] to show more examples of diversity". This was an easy change for us to make to our submission, but without the feedback form we never would have known this was something we were lacking or that the judges were stressing. There was no other way of knowing this.

At its core FIRST is an educational program for students. We want to see the students learn as much as possible through their participation in the program. In the classroom the feedback loop between teachers and students via the grading process is integral to the learning experience of the students. Especially in courses that focus on writing and presenting skills, students thrive on feedback from their teachers which allow them to develop and improve their skills over time. The Chairman’s process should be no different. We’ve all witnessed numerous students evolve by being part of a Chairman’s submission. They’ve become talented writers and presenters, and used these skills to earn scholarships and job offers at the top universities and companies across the world. But by eliminating the feedback form, we’ll be depriving these students of an incomparable experience. Where else other than FIRST do students have the opportunity to get feedback on their presenting skills from executives from Fortune 500 companies and leaders in our community? The Chairman’s feedback form represents a unique facet of the FIRST® program that we should not let go of.

Consider the analogy to the classroom; imagine that students work for an entire semester on major paper and presentation. After everything has been submitted and evaluated, the teacher steps up to the front of the classroom and announces the name of the student with the best overall submission. No one else is given any grades, feedback, or comments on their work. This would leave the other students feeling slightly dumbfounded and definitely with no sense of how their work was received or what they should improve in the future.

Quote:

Originally Posted by FRC Blog
In order to provide teams with feedback that is effective and most useful, we will offer the following ways teams can get feedback for the 2015 season…
...
Your team will get an extra 2 minutes in the CA room with the judges for extra presentation.

While 2 extra minutes of presenting time will be a welcome relief for many teams, this comes at the expense of 2 minutes of question time with the judges. This means the actual interaction with the Judges, a real avenue for feedback, will be cut down to 3 minutes for teams who choose this approach. This provides even less feedback for teams.

Quote:

Originally Posted by FRC Blog
We will continue to allow a mentor into the CA interview room for silent observation.

(This is huge – what better feedback than having your mentor, who knows you so well, listen not only to what your team is saying but especially to the questions the judges are asking. These questions are the best feedback you will ever get. Your mentors know your team and are able to give you objective and accurate feedback. Please encourage your mentor to be in the room!)

Leaving aside that the current rules only allow you have a mentor in the room at the expense of a presenting student, I don't think this is a particularly useful avenue for feedback. By the time of the actual presentation, the mentor will have seen the presentation over a hundred times. They have had plenty of opportunity to offer feedback. Yes, this gives them a chance to give the presenters feedback on how they answered the questions, but that's all. The mentors don't know and understand the Chairman's criteria they way the judges do, their feedback is just conjecture based on their best understanding of the award. For younger teams who are new to this process, that level of understanding can be tenuous at best. Regardless, this only addresses the presentation. Where's the feedback on the essay coming from? Yes teams can have the essay read by teachers and mentors, but that's still not going to help them understand what the judges are looking for.

Quote:

Originally Posted by FRC Blog
And take advantage of the increased opportunity to get real-time feedback from your judges and feedback from each other.

I fail to see how removing the feedback form and allowing teams to present for longer at the expense of the question period has increased the opportunity for real-time feedback from the judges. In fact, I think it's pretty clear that there will be substantially less room for feedback this season.

Giving proper feedback takes time. I fully understand that our judges are volunteers who are extremely busy. With some events having upwards of 40 Chairman's Submissions, giving feedback to all teams based on their entire submission must be an incredibly time consuming and difficult process. It is definitely asking a lot of the judges to have to complete these forms. At the same time the Chairman's Award is FIRST's most prestigious and important award. If we're going to hold this award as high as we do, and expect teams to hundreds if not thousands of people-hours into their efforts and submissions, I don't think it's too much to ask that they still receive a short feedback form that summarizes their strengths and weaknesses. If the check box rubric is too rigid and quantitative for tastes of some, then do away with it. Go with free form text boxes for strengths and weaknesses. It's still essential that teams get back some comments from the judges to help them know how to improve, not just when it comes to Chairman's submissions, but also in their own personal development of their writing and presenting skills.

If the workload is the issue, perhaps we can remove the expectation that feedback forms be handed back at the end of the event. Perhaps they can be emailed to the teams a few days after the event. Maybe we could have one person just do feedback on the essay (which could be done long before the event and only done once a season), and one person do feedback on the presentation. Thereby splitting and lessening the load on the feedback writers. I'm sure as a community we could come up with numerous ideas to streamline this process.

Amanda Morrison 22-01-2015 19:54

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Karthik (Post 1432378)
If the workload is the issue, perhaps we can remove the expectation that feedback forms be handed back at the end of the event. Perhaps they can be emailed to the teams a few days after the event. Maybe we could have one person just do feedback on the essay (which could be done long before the event and only done once a season), and one person do feedback on the presentation. Thereby splitting and lessening the load on the feedback writers. I'm sure as a community we could come up with numerous ideas to streamline this process.

Actually, helping to brainstorm for ideas to assist FIRST in this burdensome task is a great idea. I like the idea of separate essay & presentation feedback, for instance.

One thing that crossed my mind - a "court reporter" type position that can take the judges' quick notes on a team's essay and presentation and type it up on behalf of the judges. This could be available after the event for teams that want to retrieve it.

I see it working this way: The reporter is not a judge but merely taking notes. After a team's presentation, the judges can discuss quickly among each other and communicate a few small things about the team's presentation, and pass that along to the reporter (or the reporter can be present while they are talking). The reporter types this up in a standard feedback form, and when finished, these forms are provided to Pit Admin in an envelope for teams to pick up.

There are probably a few issues with this idea. Privacy is one that immediately comes to mind. But perhaps this volunteer doesn't necessarily even need to be in the room during the presentations. If the judges keep track of the order in which teams present, the reporter can assign a letter or number on the feedback pages, and the Judge Advisor can stuff the appropriate team's feedback into their already-numbered envelope. The reporter never sees more than an A-B-C letter.

Another issue is that this is a somewhat trusted and difficult position to fill. You need someone that is professionally literate and able to communicate the judges' feedback - but maybe this would be a great entry position for a corporate volunteer with little FIRST experience. Someone that types quickly and would be able to meet that strict deadline is also helpful. :)

This feedback might not be more than a couple sentences, but it is something.

Anyway. A quick idea, perhaps not the best, but the first thing that came to mind. Feel free to expand or poke holes through it. What else can we come up with, CD?

Andrew Schreiber 22-01-2015 20:15

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Amanda Morrison (Post 1432407)
What else can we come up with, CD?

I've been meaning to post this paper for about a week or so, but it's a self eval form a few of us put together.

http://www.chiefdelphi.com/media/papers/3095

PayneTrain 23-01-2015 00:04

Re: [FRC Blog] Chairman’s Award Feedback
 
I wanted to just let this go, but since the soapbox seal has been justly broken and I still find the decision... perplexing... I'll fire up my old soapbox for the first time in a while.

In the sixth grade, I had a science teacher that had a lesson the first day to help refresh the idea of the scientific method in our young minds. The tool to help us was a menacing black wooden crate casting a shadow from the front table onto the floor below, affectionately called "Die Wunderbox". The question: what would happen if he poured clear water through a funnel in the box? We took time observing the box, and due to its large nature, its rusting MasterLock keeping its contents hidden from us, and the teacher's general wackiness, we hypothesized that the squirrely tube at the base of the box would output something different into a glass beaker on the floor, transforming the clear water he would pour into a funnel on the top. Our hypothesis was supported by the test. Clear water went in, and a lot of dyed water came out. Through this single input into the black box, we received a different output. How did the box change the input on the way to its output? He promised he would show us on the last day of eighth grade.

Almost 3 years later, June rolls around and the whole 8th grade class of around 100 students crowds Mr. Longworth's room to see him reveal to us the inner workings of the mystery black box. For years we wanted to see a peek behind the curtain, a hint of what it could be. He pulls the key out of his pocket, jiggles it into the lock, and turns all the tumblers. The air is still in the room as the door swings open. Then we finally all see the secret...

It was a bright haired, wrinkly plastic troll doll, grasping a sign reading "Nice try, Suckers!" After all the anticipation, after youthful curiosity had reached its zenith, we literally got trolled. He took out the mechanism that gave us our output, for no reason other than a couple laughs at anguished students at the height of puberty.

FRC judging is kind of like the menacing, splintery black box. Teams spend all year preparing for a couple sessions where they can provide input to the judges. For all judged awards except one, the inputs of time and effort into executing robot design and growing the team's outreach and reaching organizational milestones were funneled into the black box all at once, and out came all the awards winners. All except for one. The "black box" for the Chairman's Award was never fully clear, but it was a far cry from the black box. Through its fuzzy, translucent shell one could reasonably see shadows of how their inputs of the written submission, presentation, and Q&A would generate a slightly clearer output than one you would get from the splintery, opaque box used for all the other awards.

422 was able to take the output of feedback from the judges over a 3 year period and turn a total dud of a team into a good team with a total dud of a submission, into a team that took home the award for the first time in its 15 years. Still, we kept trying to pry open the box; cut the lock, break the glass, just see what was really happening inside the Wonder Box of Chairman's decision making. Instead, we come back later to see the box ajar, ready to have its lid peeled open and reveal...

a troll... a bright haired, wrinkly plastic troll doll, grasping a sign reading "Nice try, Suckers!"

Jacob Bendicksen 23-01-2015 01:09

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Andrew Schreiber (Post 1432416)
I've been meaning to post this paper for about a week or so, but it's a self eval form a few of us put together.

http://www.chiefdelphi.com/media/papers/3095

I like this form. It's more specific than the form that FIRST used up until this year, and it ditches the range of 10 checkboxes for each criterion for three checkboxes and a place for comments. In my experience, people pretty much took a 9 the same as a 10 on those forms, so this provides an opportunity for the judges to be more specific. I also like the spot for presentation-specific feedback, since that's the one aspect of your submission that you can change in between events.

Justin Montois 23-01-2015 07:35

Re: [FRC Blog] Chairman’s Award Feedback
 
From Franks section of the blog post..."We have received some feedback about these changes from individual teams, as well as our Hall of Fame teams, expressing concern about the lack of written feedback for teams this year"

From the judges note in the blog post...."Teams learn best from each other.

(Your team can learn so much from the Hall of Fame teams)"

Maybe we should listen to the Hall of Fame teams then. Like Karthik and Amanda showed, there are solutions. Give the HoF teams a chance to have some input on the matter.

rsisk 23-01-2015 11:01

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by vhcook (Post 1432279)
Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

That doesn't seem like a very positive change, maybe the Admin Manual is in error, keep an eye out for a possible update.

Karthik 23-01-2015 11:21

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by rsisk (Post 1432660)
That doesn't seem like a very positive change, maybe the Admin Manual is in error, keep an eye out for a possible update.

Agreed, I'm assuming this is just an oversight based on what they said in the blog.

Jon Stratis 23-01-2015 11:22

Re: [FRC Blog] Chairman’s Award Feedback
 
I definitely think the option of a silent mentor attending should not eliminate a presenter. It kinda sorta made sense before when it was introduced, but removing the feedback form means we have to make other changes as well, and this would be one I would expect. I would also like to see this optional mentor position be opened up to more than just a team mentor. Allow any option silent observer.

This would allow teams (hall of fame, or teams that seem to always win a regional chairman's award, etc) to offer up their services to others at their regional to provide feedback. I think even more important than feedback from the judges is feedback from the teams who have already achieved what your striving for. They know how much work it is, they know what they did and probably have a good idea of what you do (how many judges follow other team's Twitter and Facebook accounts to keep up to date on local activities?), and they probably know and understand any limitations your team is dealing with. They can tell you what areas you are weak in, and highlight items that you may have done that didn't come across in the presentation. Unlike judges, they can be around to help you team practice and improve in the off season, or able to sit in before the event next year to provide extra feedback. And on the plus side, they haven't already seen the presentation a half hundred times!

philso 23-01-2015 13:03

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Jon Stratis (Post 1432673)
I definitely think the option of a silent mentor attending should not eliminate a presenter. It kinda sorta made sense before when it was introduced, but removing the feedback form means we have to make other changes as well, and this would be one I would expect. I would also like to see this optional mentor position be opened up to more than just a team mentor. Allow any option silent observer.

This would allow teams (hall of fame, or teams that seem to always win a regional chairman's award, etc) to offer up their services to others at their regional to provide feedback. I think even more important than feedback from the judges is feedback from the teams who have already achieved what your striving for. They know how much work it is, they know what they did and probably have a good idea of what you do (how many judges follow other team's Twitter and Facebook accounts to keep up to date on local activities?), and they probably know and understand any limitations your team is dealing with. They can tell you what areas you are weak in, and highlight items that you may have done that didn't come across in the presentation. Unlike judges, they can be around to help you team practice and improve in the off season, or able to sit in before the event next year to provide extra feedback. And on the plus side, they haven't already seen the presentation a half hundred times!

That is an excellent idea. Allowing teams to be critiqued by their (more accomplished) peers.

MamaSpoldi 23-01-2015 13:05

Re: [FRC Blog] Chairman’s Award Feedback
 
I too find this removal of feedback disturbing... and their justifications sadly lacking. I also concur that the ideas mentioned so far regarding (1) feedback on the written submission which could be done once before competition as well as the idea of (2) assigning someone to serve as a "court reporter" to type up individual team feedback. These are really good and helpful ideas and if we continue working together on this topic I suspect we can come up with more.

I do have one one question that I have not heard yet...
Having judged at many FLL tournaments both local and state I am intimately familiar with their scoring rubrics and the judging process used at that level. And through that process I have learned that the perception of individuals viewing the same materials and hearing the same facts can vary widely based on their personal experience and area of expertise. Don't the FRC Chairman's judges use some sort of objective scoring system (eg. a rubric of sorts) to narrow down the field for discussion during their deliberation process? It seems to me that providing access to that score sheet at least provides some specific feedback to the team on how well their message was communicated and therefore if their presentation/submission was received as intended. Providing at least this portion of feedback seems like it would require almost no effort since these scores are being accumulated anyway. Maybe if we could combine this "low cost" rubric-like numeric scoring feedback with the somewhat off-loaded "court reporter" free-form subjective feedback, it seems like we would be doing everyone in the community a service.

Providing no feedback what so ever creates a process that lacks transparency. This can breed the appearance of favoritism and unfairness because there is no evidence refute it. This is not the culture change we are all striving for.

Taylor 23-01-2015 13:09

Re: [FRC Blog] Chairman’s Award Feedback
 
Having served as a judge for an FTC qualifying event recently, I was heavily disappointed that I was not allowed to give constructive feedback to teams.

AllenGregoryIV 23-01-2015 13:17

Re: [FRC Blog] Chairman’s Award Feedback
 
To sort of add to this discussion. Surprisingly some of the best feedback we have ever received came from judges at the two events where we won the Chairman's Award. The feedback on the form was helpful, but in both cases members of the judging team talked directly with either myself or a few of the team members and gave us feedback. Some of it was just congratulatory but part of it was talking about things they liked and also a little about what distinguished our submission from the others at the event and advice on how to improve for Championships. This clearly can't be done for every team but to me we should be trying to get closer to that level and not moving away from it.

ehochstein 23-01-2015 13:18

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Taylor (Post 1432766)
Having served as a judge for an FTC qualifying event recently, I was heavily disappointed that I was not allowed to give constructive feedback to teams.

Come up to Minnesota! We give feedback to all FLL and FTC teams here :)

Libby K 23-01-2015 13:23

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by ehochstein (Post 1432781)
Come up to Minnesota! We give feedback to all FLL and FTC teams here :)

I'm in the middle of editing a much longer-form response in this thread, but I was just about to comment something similar with regards to FLL/FTC feedback: When I judged in Northern NY, we gave feedback forms in addition to comments.

So why are those programs not standardized? How is that fair to the teams? Why, if different regions can decide to do whatever they want for the younger programs, does FRC lock down any and all chances at team feedback?!

Something is seriously not right here.

(P.S. I sincerely hope that HQ doesn't respond to this by taking feedback away from all programs. That's not my intent in commenting about it. I'm much more on the side of feedback in as many ways as possible, but like I said - much longer post incoming later.)

PayneTrain 23-01-2015 13:29

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Libby K (Post 1432786)
I'm in the middle of editing a much longer-form response in this thread, but I was just about to comment something similar with regards to FLL/FTC feedback: When I judged in Northern NY, we gave feedback forms in addition to comments.

So why are those programs not standardized? How is that fair to the teams? Why, if different regions can decide to do whatever they want for the younger programs, does FRC lock down any and all chances at team feedback?!

Something is seriously not right here.

(P.S. I sincerely hope that HQ doesn't respond to this by taking feedback away from all programs. That's not my intent in commenting about it. I'm much more on the side of feedback in as many ways as possible, but like I said - much longer post incoming later.)

I'm not saying this to pile on to anyone or anything, but other than the names and general award qualities/quantities, I'm not really convinced there is true, enforced standardization at all FRC events in the judging process, whether or not it's intended or official.

Taylor 23-01-2015 13:35

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by ehochstein (Post 1432781)
Come up to Minnesota! We give feedback to all FLL and FTC teams here :)

It was disallowed at the HQ level.
Quote:

Originally Posted by FTC Judge Manual, page 19
Sometimes a Team may request feedback from Judges once the Event is over, and the Judges must not provide feedback to Teams

This especially stuck out to me because I missed this on the Judge Certification (as did other FTC judges at that event with illustrious backgrounds in FRC)

Andrew Schreiber 23-01-2015 13:38

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Taylor (Post 1432799)
It was disallowed at the HQ level.

This especially stuck out to me because I missed this on the Judge Certification (as did other FTC judges at that event with illustrious backgrounds in FRC)

Hang on hang on, there's a Judge Cert for FTC, there's not for FRC :\

Taylor 23-01-2015 13:39

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Andrew Schreiber (Post 1432801)
Hang on hang on, there's a Judge Cert for FTC, there's not for FRC :\

It was optional.

Libby K 23-01-2015 13:41

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Taylor (Post 1432799)
It was disallowed at the HQ level.

This especially stuck out to me because I missed this on the Judge Certification (as did other FTC judges at that event with illustrious backgrounds in FRC)

What was your Judge Cert process like? Ours was a ~20 minute powerpoint review before opening ceremonies.

As noted from this tiny sidetrack on the main discussion, there are so many -other- things that should be corrected surrounding judging... I think this gigantic step in the wrong direction should not have been the first on the priority list.

(Again - long-form response coming later. I've got a day job now...)

Andrew Schreiber 23-01-2015 13:41

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Taylor (Post 1432804)
It was optional.

I don't suppose it's in a format you could share? I'm curious.

Taylor 23-01-2015 13:52

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Libby K (Post 1432806)
What was your Judge Cert process like? Ours was a ~20 minute powerpoint review before opening ceremonies.

Quote:

Originally Posted by Andrew Schreiber (Post 1432807)
I don't suppose it's in a format you could share? I'm curious.

It was an online quiz. I'm not comfortable sharing the link; I got the distinct feeling that it is not for public consumption.
I think the single sentence I quoted from the judge manual was stretching things as it is. As the manual isn't posted publicly (to my knowledge) I'll choose to keep it that way.
With the disclaimer that I believe that if HQ is indeed not allowing feedback, the judge manuals really ought to be available to teams. But that's not my call to make, and they have their reasons.
FTC teams are encouraged to use the Self Reflection Sheet; I haven't seen a version for FRC.

ehochstein 23-01-2015 14:46

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Taylor (Post 1432799)
It was disallowed at the HQ level.

I know, fortunately both I and the MN partner disagrees with that.

Quote:

Originally Posted by Libby K (Post 1432786)
So why are those programs not standardized? How is that fair to the teams? Why, if different regions can decide to do whatever they want for the younger programs, does FRC lock down any and all chances at team feedback?!

Having been at a few meetings where feedback (for FTC) was discussed, there are a few partners that are very adamant that feedback is not given to teams because teams then complain to them (the partner) about it. We don't experience that in Minnesota but other areas do. My theory is that it has to do with the different cultures in different areas of the US.

Navid Shafa 23-01-2015 14:58

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by ehochstein (Post 1432781)
Come up to Minnesota! We give feedback to all FLL and FTC teams here :)

We do the same in Washington.

Thad House 23-01-2015 15:25

Re: [FRC Blog] Chairman’s Award Feedback
 
So with the admin update today, it looks like the mentor can be in addition to the 3 students. That should help some of the issues of not having feedback.

http://www.usfirst.org/sites/default...Bundle0123.pdf

Hot_Copper_Frog 23-01-2015 17:01

Re: [FRC Blog] Chairman’s Award Feedback
 
I realize that judge feedback is not always perfect, and can be upsetting. But good gracious, it is far better to be upset by something that you know, than to be upset with what your imagination is fabricating inside your skull. Doesn't FIRST teach us that knowledge is power? I just don’t see how this decision is in the best interests of students trying to learn and improve. It seems like a decision that causes less trouble at an administrative level.

As a project judge for FLL, I can admit that providing fair, constructive feedback for every team was a challenge. It’s time consuming during and already time crunched day, and sometimes I've just been at a loss for words. But to a degree, it keeps you honest and fair, and gives you a solid place to “negotiate” with other judges. By taking away all feedback, FIRST is turning the Chairman’s Award into a blackbox that allows for an uncomfortably low level of transparency.

Normally I can at least see the merits of both sides of a controversial FIRST policy change. But the more I think about this and discuss it with other mentors, the more I’m convinced that it’s not in the best interests of the students.

Alex2614 10-03-2015 21:34

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Thad House (Post 1432857)
So with the admin update today, it looks like the mentor can be in addition to the 3 students. That should help some of the issues of not having feedback.

http://www.usfirst.org/sites/default...Bundle0123.pdf

This is correct. THIS makes me more okay with the change. Allowing the mentor in the room IN ADDITION TO THE 3 presenters is key. Having gone through the chairman's interview process several times, both as a student, and now as a mentor, I have seen the highs and lows of the feedback forms. 99% of the time, they were NOT helpful at all. They never told the kids what could have been done or demonstrated better, and they were never constructive. And half the time, we were docked for things that were clearly demonstrated, communicated, and explained in the room, binder, video, essay, etc. Still docked for it. There were some events that we never actually got our feedback in the first place. Having the mentor in the room, and receiving comments from HoF, other teams, and other judges will help a lot.

We shall see. FIRST is trying something out. FIRST has changed their minds "after the fact" before due to overwhelming complaints from teams. It's not unprecedented, and if this doesn't work in the way FIRST hopes it will, it can be changed back.

Jacob Bendicksen 10-03-2015 23:44

Re: [FRC Blog] Chairman’s Award Feedback
 
As someone who presented Week 1, having the mentor in the room was somewhat helpful.

It's better than not getting any feedback at all, but he wasn't able to offer more than 'they really liked you and were very interested in that thing' because he can't read their minds.

We won Chairman's at that event, and we don't really know what could be improved for District Champs, since we didn't get any direct feedback from the judges. Sure, having the mentor in there is better than no feedback whatsoever, but it's not very helpful in terms of improving the presentation for subsequent events. I think the feedback form should come back next year - a mentor, as wonderful as they are, doesn't know what the judges were thinking, and it's way harder to improve without it.

wilsonmw04 10-03-2015 23:53

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Jacob Bendicksen (Post 1456314)
As someone who presented Week 1, having the mentor in the room was somewhat helpful.

It's better than not getting any feedback at all, but he wasn't able to offer more than 'they really liked you and were very interested in that thing' because he can't read their minds.

We won Chairman's at that event, and we don't really know what could be improved for District Champs, since we didn't get any direct feedback from the judges. Sure, having the mentor in there is better than no feedback whatsoever, but it's not very helpful in terms of improving the presentation for subsequent events. I think the feedback form should come back next year - a mentor, as wonderful as they are, doesn't know what the judges were thinking, and it's way harder to improve without it.

I can't wait for this pilot year to be over. the lack of any feedback boggles my mind.

PayneTrain 11-03-2015 00:23

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by wilsonmw04 (Post 1456317)
I can't wait for this pilot year to be over. the lack of any feedback boggles my mind.

Hopefully the judges will at least let a mentor in for VA so if y'all don't win you have something for Chesapeake. We don't have squat :/

Alex2614 11-03-2015 02:48

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Jacob Bendicksen (Post 1456314)
As someone who presented Week 1, having the mentor in the room was somewhat helpful.

It's better than not getting any feedback at all, but he wasn't able to offer more than 'they really liked you and were very interested in that thing' because he can't read their minds.

We won Chairman's at that event, and we don't really know what could be improved for District Champs, since we didn't get any direct feedback from the judges. Sure, having the mentor in there is better than no feedback whatsoever, but it's not very helpful in terms of improving the presentation for subsequent events. I think the feedback form should come back next year - a mentor, as wonderful as they are, doesn't know what the judges were thinking, and it's way harder to improve without it.

I think if it does come back, it needs to be overhauled from what it was before. You couldn't really tell what the judges were thinking with the old system either. I think it should be more qualitative than quantitative. There should be a comment section for each "category" for the judges to explain WHY they think what they do. Before it was just a ranking from 1-10, but no way of knowing why a judge chose the number that they did. Because, again, we've had it happen several times where they dock us for something, even though it is explained in every part of the submission.

So, in closing. If it comes back, t should be mostly qualitative comments on why the judges chose the way that they did, and actually talk about things that could be improved, rather than an arbitrary numerical ranking that can completely mislead and can change from team to team as the day goes along, included with only a couple lines for random comments. It should be completely opposite.

Alex2614 11-03-2015 02:51

Re: [FRC Blog] Chairman’s Award Feedback
 
Also, I really look forward to hearing from other teams with this new presentation video we are supposed to submit, and more importantly I look forward to seeing all of the other winning presentations from this year. Honestly seeing what all the other presentations look like will be more valuable than any judge feedback form would ever be. The feedback forms usually left us more frustrated and confused than enlightened. By actually seeing what the other teams did will be inspiring and educational. So whatever happens, THAT needs to stay. The presentation video is a change that I welcome wholeheartedly.

RoboMom 11-03-2015 07:40

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by ehochstein (Post 1432781)
Come up to Minnesota! We give feedback to all FLL and FTC teams here :)

The FTC program specifically asked that judges not give feedback this year. FTC also publishes the Judges Manual on a site available to everyone.

http://www.usfirst.org/roboticsprogr...teer-resources

Andrew Schreiber 11-03-2015 07:52

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Jacob Bendicksen (Post 1456314)
As someone who presented Week 1, having the mentor in the room was somewhat helpful.

It's better than not getting any feedback at all, but he wasn't able to offer more than 'they really liked you and were very interested in that thing' because he can't read their minds.

We won Chairman's at that event, and we don't really know what could be improved for District Champs, since we didn't get any direct feedback from the judges. Sure, having the mentor in there is better than no feedback whatsoever, but it's not very helpful in terms of improving the presentation for subsequent events. I think the feedback form should come back next year - a mentor, as wonderful as they are, doesn't know what the judges were thinking, and it's way harder to improve without it.



Take a look at the feedback form I posted earlier (http://www.chiefdelphi.com/media/papers/3095). Having judged feedback is nice but I think having a third party (perhaps an English teacher at your school) fill out some sort of feedback (to avoid bias) will likely help you iterate.

And I still agree with you, lack of feedback is terrible.

wilsonmw04 11-03-2015 09:27

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Alex2614 (Post 1456355)
So, in closing. If it comes back, t should be mostly qualitative comments on why the judges chose the way that they did, and actually talk about things that could be improved, rather than an arbitrary numerical ranking that can completely mislead and can change from team to team as the day goes along, included with only a couple lines for random comments. It should be completely opposite.

Even the flawed sheet we have received in the past is better than nothing at all. I am sensitive to the judges time. The tool should be take less than 5 minutes to complete per team. That should be more than enough time to give constructive feedback.

g_sawchuk 11-03-2015 09:36

Re: [FRC Blog] Chairman’s Award Feedback
 
Interesting. I strongly dislike how having a mentor takes away from having 3 student presenters. In this case, I feel that it is best to have 3 student presenters. Your students are ideally what have made you worthy for the CA, and therefore you should have as many students as possible to showcase your accomplishments.

BigJ 11-03-2015 09:38

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by GrifBot (Post 1456411)
Interesting. I strongly dislike how having a mentor takes away from having 3 student presenters. In this case, I feel that it is best to have 3 student presenters. Your students are ideally what have made you worthy for the CA, and therefore you should have as many students as possible to showcase your accomplishments.

You can have a silent observer in addition to 3 presenters starting this year.

Libby K 11-03-2015 09:45

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by GrifBot (Post 1456411)
Interesting. I strongly dislike how having a mentor takes away from having 3 student presenters. In this case, I feel that it is best to have 3 student presenters. Your students are ideally what have made you worthy for the CA, and therefore you should have as many students as possible to showcase your accomplishments.

That was last year. This year, a mentor may join your three presenters as a silent observer.

Something interesting happened at Mt. Olive this weekend - the JA invited one mentor and one Chairman's presenter for an all-team 'meeting'. It wasn't exactly a feedback session, but they handed back everyone's USB drive & forms, and then shared a few tips that they noticed from this round of presentations. (Things like, remember that setup time is included in your minutes, etc.)

Their first one: Make sure your mentor-observer has a good poker face. It'll totally throw off your students, and could (potentially) be seen as coaching if your mentor is actively reacting to the things your students say.

Monochron 11-03-2015 11:12

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by BigJ (Post 1456412)
You can have a silent observer in addition to 3 presenters starting this year.

Just hope that you are lucky and your Chairman's Judges have read the manual thoroughly. If not they will argue that you can only 2 students presenting if a mentor is observing.

This is what happened at Palmetto and it was very unfortunate. As our first time presenting for the Chairmans award it would have been really nice to have some sort of feedback as to how it went. We think we have a lot to improve on, but without anyone taking notes in the room, we really can't be sure of details.

Andrew Schreiber 11-03-2015 11:15

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Monochron (Post 1456458)
Just hope that you are lucky and your Chairman's Judges have read the manual thoroughly. If not they will argue that you can only 2 students presenting if a mentor is observing.

This is what happened at Palmetto and it was very unfortunate. As our first time presenting for the Chairmans award it would have been really nice to have some sort of feedback as to how it went. We think we have a lot to improve on, but without anyone taking notes in the room, we really can't be sure of details.

If this is the case have them talk to the Judge Advisor at the event. It was stressed in their training that change came.

Monochron 11-03-2015 12:35

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Andrew Schreiber (Post 1456461)
If this is the case have them talk to the Judge Advisor at the event. It was stressed in their training that change came.

They were informed eventually (maybe after judging ended), but we couldn't put our presentation on hold while we located the Advisor, got him to talk to them, and got them to agree. The schedule was tight and delays weren't an option. In addition to that I believe there was concern about changing what was allowed after multiple teams had already presented.

We didn't know there was an issue until we actually went in to present. It became public knowledge by the end of the day, but there was little more that we could do than to protest to the Administrators.

ehochstein 11-03-2015 12:48

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by RoboMom (Post 1456373)
The FTC program specifically asked that judges not give feedback this year. FTC also publishes the Judges Manual on a site available to everyone.

http://www.usfirst.org/roboticsprogr...teer-resources

From a volunteer's point of view, my goal is to support the teams in Minnesota. The teams in Minnesota have expressed to me multiple times how horrible it would be to not receive feedback at an event, as it improves their team. I also want the Minnesota teams that move on to the Super Regional to be able to move on to Championships. If I can give these teams feedback in order to improve the quality of the program and increase the chances of that team moving on, I'm going to do it.

As a coach of a FTC team, I also want feedback. It helps me tell the students what they did, where they can improve and what they should change. It also helps the students realize this for themselves. As someone that volunteers at every single FTC event in Minnesota, I don't spend events with my team and if it were not for the feedback I would be clueless as to how my team did in judging.

It blows my mind that feedback isn't allowed to be provided at both the FTC and FRC level.

Disclaimer: Giving feedback to teams at events in MN is not up to me, I don't make that decision, I am just an incredibly strong supporter of feedback.

Qbot2640 11-03-2015 12:55

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Monochron (Post 1456482)
They were informed eventually (maybe after judging ended), but we couldn't put our presentation on hold while we located the Advisor, got him to talk to them, and got them to agree. The schedule was tight and delays weren't an option. In addition to that I believe there was concern about changing what was allowed after multiple teams had already presented.

We didn't know there was an issue until we actually went in to present. It became public knowledge by the end of the day, but there was little more that we could do than to protest to the Administrators.

I had the same experience at Palmetto, and began to question the situation until the judge stated that all teams had been held to that same criterium. As much as I wanted to be present, I was not going to push for something that was not allowed to teams presenting earlier. For our developing program, it was a significant disappointment, however, as I really wanted to help the presenting group improve for our next regional, and for next year.

Qbot2640 11-03-2015 12:57

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Libby K (Post 1456418)
Something interesting happened at Mt. Olive this weekend - the JA invited one mentor and one Chairman's presenter for an all-team 'meeting'. It wasn't exactly a feedback session, but they handed back everyone's USB drive & forms, and then shared a few tips that they noticed from this round of presentations. (Things like, remember that setup time is included in your minutes, etc.)

That would have been great...especially considering that with the excitement of eliminations, my team (and I'm sure many others) forget to go back and get the USB drive...I've lost a few this way.

AllenGregoryIV 11-03-2015 13:36

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by Libby K (Post 1456418)
Their first one: Make sure your mentor-observer has a good poker face. It'll totally throw off your students, and could (potentially) be seen as coaching if your mentor is actively reacting to the things your students say.

This is one of the reasons myself and several other coaches I know won't go into the presentation room. Being in the room isn't a replacement for feedback. I've seen the presentation dozens of times and I think it's awesome, I wouldn't put the students in the room unless I though it was awesome. Other than reading the judges faces, what feedback am I supposed to be able to take away from being in the room with the students, I really don't understand it. In my opinion the only people that can provide accurate feedback are the judges making the decision.

mjc49 11-03-2015 14:06

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by ehochstein (Post 1456486)

As a coach of FTC team, I also want feedback. It helps me tell the students what they did, where they can improve and what they should change. It also helps the students realize this for themselves. As someone that volunteers at every single FTC event in Minnesota, I don't spend events with my team and if it were not for the feedback I would be clueless as to how my team did in judging.

It blows my mind that feedback isn't allowed to be provided at both the FTC and FRC level.

Disclaimer: Giving feedback to teams at events in MN is not up to me, I don't make that decision, I am just an incredibly strong supporter of feedback.

Having feedback as a coach of Rookie FTC teams and having our FRC team submit a Chairman's Award for the first time this year is/would be very valuable. Allowing a mentor to sit in on the presentation was very helpful. I'm a firm believer in feedback.

Going through this process for the first time, we have a lot to work on so it's easy to identify areas to improve on. I expect as we mature, having the feedback of how the judges interpreted our presentation would be much better if received it from them vs. trying to figure it out on our own.

Alex2614 11-03-2015 14:51

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by GrifBot (Post 1456411)
Interesting. I strongly dislike how having a mentor takes away from having 3 student presenters. In this case, I feel that it is best to have 3 student presenters. Your students are ideally what have made you worthy for the CA, and therefore you should have as many students as possible to showcase your accomplishments.

This is NOT the case anymore. The rules have been changed prior to week 1. You can have up to 3 presenters in ADDITION to a mentor as a silent observer.

RoboMom 19-03-2015 20:05

Re: [FRC Blog] Chairman’s Award Feedback
 
Feedback is a multi-faceted topic. I want to mention an excellent read - "Thanks for the Feedback - the science and art of receiving feedback well" by Douglas Stone and Sheila Heen. I've had quite a bit of training on feedback through my job the past couple of years and see the pros and esp. the cons if it is not done well and done appropriately for the circumstances. It is a topic that is much mis-understood.

YAK'ker 19-03-2015 21:46

Re: [FRC Blog] Chairman’s Award Feedback
 
Quote:

Originally Posted by vhcook (Post 1432279)
Wow. I thought allowing a silent mentor in the room was going to be in addition to the three student presenters. That is definitely a negative change given the removal of feedback. I should have read this bit of the manual more carefully. I also see that if you extend your presentation, it cuts into Q&A time.

So - we had 3 students and one silent mentor in the room (me.) This is per the admin rules, page 42, that states: "Not more than three (3) student team members, plus one (1) adult mentor, are allowed to attend the interview."


All times are GMT -5. The time now is 17:42.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi