A solution for Animation Judging for 2005

[TKatsAniMentor]

Hello All,

I am one of the sponsors for the TechnoKats (45). We, myself and the other sponsors, have to say that the judging and overall enthusiam for the animation competition is at an all time low. Below are some comments and suggestions that we have discussed about the competition.

Regional judging still has problems. Not every regional has the same amount of teams attending; Therefore, teams at the least attended regionals have a much better chance of winning the award. After checking the release from Autodesk, we noticed that the NYC Regional has only 3 animations to be judged, while the Great Lakes Regional had 27 entries. Also, varying judging conditions and locations have also been a problem. Until these discrepancies are addressed, regional judging will not be "fair."

Another thing that needs to be addressed is the support and involvement of Autodesk with the animation competition. Autodesk does supply software to all the teams to create an animation. Does anyone use this software? We use AutoCad, 3DS Max, Adobe Premiere, Adobe PhotoShop, Macromedia Flash, Sound Forge, and Acid to do our animations. Only 1 piece of our software list is provided. I have heard a nasty rumor that Autodesk is going to force us to only use what is provided or the entry may be DQ'ed.

2002 and before Autodesk had a group of industry professionals that graded animations on Technical Execution, Content, and Creativity. These areas had their own awards as well as the overall and rookie awards. In 2003, Autodesk decided it would be best to eliminate the other honorable awards and give only the overall and rookie. I have to say being the Technical Excellence Honorable Mention Award winners in 2002 that we were upset about the change. With the judging leaving the hands of professionals and it being thrown at the students, all the current problems arose. First, the animation creation time was reduced by 1 month to coincide with the robot ship date. This caused students to choose either to work on the robot or the animation (in most cases) and forced the animators to lose more sleep due to the shortened deadline. Second, judging at last year's competitions was a total joke. Lastly, the entry that won the whole thing did not even win a regional competition and was mostly video, not animation.

Now that most of the problems are now addressed, let's talk solutions.

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

Anything else you would like to add to the solutions list is welcome. Please no more gripes. There has been plenty of threads of those already. This was created to try to brainstorm solutions to fix the current problems.

Thanks for reading this.

[Joe Matt]

While I was not on animation this year, I watched and somewhat helped (ok, not much) them. The main focus of the media team was the animation, so I know where some of the problems are.


1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.


1.) I agree, but also disagree. I think that students should have an opinion too in the animation award. I think they should judge for the things they did in 2002, but then have an 'Overall score', that'll count 60% of a 100% animation score. The other 40% of the score will be done by ALL students, not just those who submitted animations.

2.) Yet again, I agree and disagree. I think that the animation should be given another month, but make it before the kickoff where you add the time. Release the theme and what you have to do, then allow for teams to use the winter break to create textures, ideas, and concepts. Then when the game comes out, lots of the work is done, and if teams want that year's robot in it, they can add it, without worrying about the other things they have been working on for a month. Ship along with the robot.

3.) Totally agree.

4.) Which answers the question, like what? A technical challenge, or a thematic challenge? :confused:

5.) At VCU we had one student sent to watch the video in a private room with a wide screen and all. I think this is good as it is.

6.) Agree again.

I don't think we'll see the animation as big as the actual competition, just like how the web site won't nor the business aspect of it. If FIRST wants to do something great, make the animation part totally separate from the robotics part. Think of it as an art contest and less of PR for your robot.

[Collin Fultz]

2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?


I totally agree with your second point. It really inhales audibly to see some students not be able to build the robot or work on the animation because they are at the same time. However, I can see why FIRST does this, and understand %100. There are some teams who don't have 24/7/365 access to computer software. This technology is only available during the six weeks. Some teams are lucky enough to have virual any-time access (234 included) and I'm thankful for that, however, I don't think it is "fair" (I hesistate to use the "f" word) to teams that don't. FIRST has reasons for everything. I see and understand this one.

The third point is what I have a main problem with. As Andy Baker said, "Don't concentrate on the entry itself, concentrate on what you do for the entry." (or something like that, sorry if I screwed it up AB) He was talking about the Chairman's Award in this quote, but it applies to all of FIRST. The Kit of Parts does not come with a rubric of what will win this year's game, nor should it. The Chairman's Award info does not come with a rubric of what is judged and what isn't, because the point isn't to win the award but to be inspired and inspiring while doing it. Cyber Blue runs a total design review process. We don't do this simply to win awards, we do it because it is a part of some fields of engineering and even if you plan on being a bum, presentation and oratory skills are vital to sucess. When students walk into the Chairman's Award presentation, they have some idea of what they are being judged upon, but I have never seen the official rubric of how I am being judged, and I'm glad for that. If there isn't a rubric for "FIRST's most prestigious award" (that's in quotes as a quote, not as sarcasm) why should there be one for anything. Once we start creating animations/chairmans awards/robots/lives simply to satisfy a rubric...the creativity is lost, the passion is gone.

I love watching animations. About all I had to do with ours was to sit and say, "Man, that's really good. Good job Ryan/Chaz." I think animations and animators are some of the coolest things in FIRST, but that doesn't make it fair. I heard somebody from our team say, "Yah, Gunn always has a cool animation." They won Arizona's Animation Award this year. Does that make it any more <shudder>fair</shudder> than when it seemed like Hammond was invincible two years in a row? "Life isn't fair."-Dean Kamen There is no letter in FIRST for Awards. The only reason I believe "Recognition" (from an award standpoint) is in there is because "FIST" isn't that inviting. Do it for the "I". Be proud of your team and your robot and your chairman's and your mentors and your animation and and and. Don't do it for the award. Again, I'm not an animator by any aspect of the word...however this stems much deeper than animation.

/climbs down from soapbox

[TKatsAniMentor]

The third point is what I have a main problem with. As Andy Baker said, "Don't concentrate on the entry itself, concentrate on what you do for the entry." (or something like that, sorry if I screwed it up AB) He was talking about the Chairman's Award in this quote, but it applies to all of FIRST. The Kit of Parts does not come with a rubric of what will win this year's game, nor should it. The Chairman's Award info does not come with a rubric of what is judged and what isn't, because the point isn't to win the award but to be inspired and inspiring while doing it. Cyber Blue runs a total design review process. We don't do this simply to win awards, we do it because it is a part of some fields of engineering and even if you plan on being a bum, presentation and oratory skills are vital to sucess. When students walk into the Chairman's Award presentation, they have some idea of what they are being judged upon, but I have never seen the official rubric of how I am being judged, and I'm glad for that. If there isn't a rubric for "FIRST's most prestigious award" (that's in quotes as a quote, not as sarcasm) why should there be one for anything. Once we start creating animations/chairmans awards/robots/lives simply to satisfy a rubric...the creativity is lost, the passion is gone.


I have to disagree with this. I am a educator and when I assign a assignment I require certain things. This does not limit my students' creativity, but helps them direct it. You said that there is no rubric with the kit of parts. I think we are talking about apples and oranges. I said what I said due to the fact that Autodesk has no consistent vision for the award. A rubric would convey their vision to all so everyone has a chance at the award.

To use the "f" word again, The whole competition will never be "fair" with the current structuring. Every team has different expertise, tools, and technology. At least with the animation, a little more direction would be nice other than just "create a commercial for FIRST."

[gsensel]

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
Totally agree, as much as the students do on an animation professional should look at it. This is so they know what truely will fly in the professional world. Many student unless they are activly immerssed in the world animation can not acuratly tell what is award worthy.

2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
Agree because it is a thrill when you see a robot that you built, in some way, competing.

3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
In schools even for little assignments a rubric is given to provide an idea, or possibly direction it is intended to go.


5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
Possibly a change for this could be a judgng that occurs before regionals, maybe later in the season so the students an do both animation and robot, is done by professionals.


6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

Totally agree, if nothing else it recognizes more than just 2 teams in a season. Also it affirms what the judges like.

[ThetaDot]

The point of student judging is to bring us into to fore a bit more. I have confidence in the kids that did the animations to be able to judge decently enough.
I had no problem with the organisation of the judging; it went pretty smooth - at least at GLR.

Also I think part of the ambiguity in what the judges want has a purpose. We are free to do what we want and to show it how we want to. If a finely detailed rubric gets distributed to us, we are going to end up seeing the same animation over and over. Personally, I made what I wanted, without regard for what would get me a win.
I agree with what Collin234 said. To do some of the things you suggest means making it less of a challenge in my mind.

Also, I do not agree with separating the animation work time from the robot build time. The visualization is an optional component and so anybody pulled between the two should obviously drop the animation and go for the robot.

This is the first year our school had done FIRST, and I don't think anybody on our team has had any complaints. I know the animation competition isn't perfect, but I am not going to ask for perfection. I am content with how it exists today.

[reisser]

The professional judging seems to be consistenly inconsistent. There must be rubric so that the "professional" judges know what to look for. I quote the word professional because certain hints in the comments we received in the past suggest that some of the national judges are not professional animators. Regardless, a set of universal criteria is necessary to make a fair and consistent judgement about a work of art.

For the past three years I have read national judging comments that contradict themselves. Some judges harp on about the "lead in" statement while another judge requested more "sexy robots". One judge commended our use of depth-of-field when there was none at all. Little inconsistencies such as these are very hard to take when we all have worked so hard. And while the piece that won last year was compelling, I have to agree that it was disappointing to see a winner picked with so little demostration of 3d animation. Again, this clarifies the need for a judging rubrik. It is to keep the judges on-task, fair, and consistent. I struggle without a clear set of criteria at regional judging. I think any judge would benefit from knowing what to look for instead of assigning arbitratry number values for a vague set of categories.

The competition documents contain hazy instructions at best. Every year there is some misunderstanding about the "credits" version and where to put a black frame. Two years ago our score sheet reported that out entry arrived late. We Fedex'd it four days early and received deilvery confirmation. Some aspects of the competition are just blatantly disorganinzed.

I have been hoping that postivie changes for the animation competition would occur. It would be really rewarding to have some good come out of our criticisms. See everyone at nationals.

[Jonathan M.]

The thing I have the biggest problem with is the deadline date. I'm the Animation leader of the team, and we couldn't even do one because the entire animation team had other things to do that concerned the robot.

I hope Autodesk doesn't limit us only to what it provided. I really like to have my other software around to help me out. Not saying that we couldn't do it with just 3Ds, but it would suck.

So yeah.. again my biggest problem is the deadline date. Hope that will change.

[Salik Syed]

I serioursly think there are problems with student judges because ...well lets just make a scenario

Person 1: They judge the animations by comparing it to pixar....(mostly low scores)
Person 2: They compare the animations to an ideal student made animation (medium scores)
Person 3: They compare other animations to their own (which lets just say isn't good) (mostly high scores)
now if person1 judges 2 & 3 but they judge 1&2, 1&3 then obviously person 1 will win!!!
now this is REALLY unfair esp. at a small animation event with like 5-6 teams so that discrepencies can't sort themselvs out

[av11d]

Yes, scoring is VERY inconsistent. Last year, we received an atrocious score from the judges in the technical portion: A 1.1 out of 20 if I remember correctly. Our animation featured character animation, bipeds, and lip syncing. We were using some of the most advanced features of 3ds max! I feel we used all those features successfully, and so did other teams, because we won 2 (or 3?) regionals.

I also agree with Salik that a scenario aforementioned could easily happen. There is no standard to base the grading off of. 0 to 100 is huge. How are you supposed to know what a 50 is? A 50 could mean average, since it's half the max score. A 50 could mean atrocious if you think about in the sense that it's a 'failing' grade. So who knows!

[kcy88]

This is really great that us animators are addressing certain issues. I was disappointed at the fact that 103's animation wasn't played when they won the award at the NJ regional. If i recall correctly, last year, they showed the winning animation on the big screen, but this year... what happened?

[SteveC116]

The four times that I have viewed the awards ceremony (this year and last year), I never saw the animation played at the end. Normally it's just a powerpoint slide.

[Ryan M.]

I agree with everything you said. However, I can understand the motivation behind limiting teams to only their product. Maybe if teams weren't allowed to use anyother commericial products, that would work. One of their reasons for limiting what products can be used is that many teams can't afford to by Adobe Premier, Flash, etc. So, if you were only allowed to use the supplied software and any freeware/open source stuff you want, maybe that would be a good compromise. :)

[RoboMom]

Animations were shown on the big screen while waiting for the awards ceremony at Chesapeake on Sat. It was great to be able to see all the entries and this should be an important part of any regional.
Unfortunately, our really hard working animator, team of one, didn't get done by deadline. I give her a huge amount of credit though. She taught herself completely, and had to work using the one licensed copy on one really lousy school computer, never once complaining. I hope she will finish it to put in her portfolio.

[Ryan Dognaux]

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

1) I think Autodesk needs to contact Disney, Pixar, etc. and get a few real professional Animators out and about at the competitions. It's like this - many students are inspired when they meet and talk to Engineers, but if we (animators) never meet Animators, then how are we supposed to get excited and be inspired?? If FIRST and Autodesk could get professionals to make the judging professional, then I'd be all for having them doing the judging. Otherwise, keep it the way it is.

2) I agree with you to an extent. I think that Animation should've been due at the same time as the Autodesk Inventor Award.

3) I think all teams should receive the scorecard ahead of time since it has clear definitions for what you will be judged for (Concept, Creativity, Technical Execution).

4) I don't even know what to think here. I think it's amazing what all of us put out each year, so it's probably a good thing to leave it the way it is.

5) Again, I think a bigger presence from Autodesk would be nice. It's almost like the few people or the person that attends the Regional is invisible.

6) I agree with you totally here. Honestly, I think Autodesk does a poor job with relaying things to teams and the animators. Autodesk has a voice, but it needs a megaphone or two to reach us..... Is it just me, or does Autodesk seem just very disconnected in this whole process? Don't get me wrong, I'm very grateful for all they do, but it just seems like we don't hear much from them.

My 2 Cents...

[dlavery]

The four times that I have viewed the awards ceremony (this year and last year), I never saw the animation played at the end. Normally it's just a powerpoint slide.

Your winning animation was played during the awards ceremonies at Annapolis.

-dave

[Lev]

3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?


The judging rubrics are (and were in all the past years) in the award rules. They are vague in some places but they actually are pretty specific.

The problem with the rubrics is what they are. I think its hard to find a more subjective category than "creativity". For example, Space and Robot scenes can be thought of as creative in a general context, but everyone in FIRST knows that this is the most common scene in FIRST animations.

Technical Execution and Content Communication are not perfect either - since these are so linked together. As an extreme example - sometimes its hard to understand whats going on in the animation simply because of poor technical execution.

Honestly, i think it would be better if there was no rubrics system at all. Even with perfectly defined rubrics, all of which make perfect sense, there will always be problem with each team having different scale - as someone already mentioned. I think a better system would be with every team just voting for best 3 (for example) animations (exluding own animation) and the one getting most votes - winning.

[SteveC116]

Your winning animation was played during the awards ceremonies at Annapolis.

-dave

Avast! The one competition I do not attend!

And about the rubrics, it doesnt matter if one team awards everyone really low scores, as long as the awards are consistant in relation to the others.

[Salik Syed]

Avast! The one competition I do not attend!

And about the rubrics, it doesnt matter if one team awards everyone really low scores, as long as the awards are consistant in relation to the others.


Well...they get a higher score as they do not rate themselves!
Them voting lower for others will only bring the OTHERS average down and not their own.

[ThetaDot]

Well...they get a higher score as they do not rate themselves!
Them voting lower for others will only bring the OTHERS average down and not their own.


so then mandate that the highest score given on your sheet is the # of points you also give your own team minus a number like 5 or 10

[Terminator6]

wells heres my little bit and then ill probably shut up and let other people figure this one out....

any way you cut it i dont think that peer judging will every be....heres the dirty f word....fair...

my analogy here is this, and if im completely mistaken sombody slap me and tell me ive been ignorant.....

the chairmans award nor any of the design awards are not decided by peer judging...so why should the same be true for animation....and i know that the point of peer judging is to allow the animators to have more control....but still

so heres my comprimise idea....i know this is crazy....but 2 awards...let one be decided by peer judging....one rep from each time like this year....and one by judges....let people know what the professionals liked and what the students liked.....

---end idea-----

[White_Orpheus]

I think the student judging is fundamentally flawed. We managed to walk away with a victory this year at the Silicon Valley regionals. While i like to think that our animation is highly sexy, we really didn't convey anything about what first does for us. Frankly in my opinion, 192 really deserved the award, but the students judged based on sexiness of the animation instead of the actual message conveyed. While it may have been flawed, i'm still glad we won ;)

[Ryan Dognaux]

so heres my comprimise idea....i know this is crazy....but 2 awards...let one be decided by peer judging....one rep from each time like this year....and one by judges....let people know what the professionals liked and what the students liked.....

That is quite possibly the best thing I've heard yet!! Seriously... the top Professional pick, and the top animator pick! Man, that is just a great idea if I've ever heard one.

Maybe Website Award should do that too :]

[rowe]

While i like to think that our animation is highly sexy, we really didn't convey anything about what first does for us.

As an animator judging the Silicon Valley regional, I graded team 253's animation very high in both creativity and technical execution but did not grade as high in message. I was very impressed with 253's animation...so congradulations (Look forward to competing against you in nationals). But back to the topic of the thread, although the student judging might not be "fair," it is also nice to have some imput. A solution for this could possible be to split the award into 2 awards, say one by judged by professionals, and the other by students, that way. Or, have a combination of student and professional judging, say 50/50 or some variation of the ratio to be worked out by Autodesk.

EDIT: oops, it looks like i didnt read the thread closely enough beacuse the 2 award idea has already been mentioned. Heh.

[Vivelation]

I think the student judging is fundamentally flawed. We managed to walk away with a victory this year at the Silicon Valley regionals. While i like to think that our animation is highly sexy, we really didn't convey anything about what first does for us. Frankly in my opinion, 192 really deserved the award, but the students judged based on sexiness of the animation instead of the actual message conveyed. While it may have been flawed, i'm still glad we won ;)

Hey, thanks! :) Our animation team works really hard, and whatever they come up with is always awesome, but your team's animation was really cool (the music and the running and the hair :D pretty neat stuff). Anyway, congrats again on your win (standing O from 192 ^^) See you guys at nationals!

[Ryan Dognaux]

Back on track here guys - I seriously think a Professional Award and a Student Award is the way to go. It would test both sides of the medium, and also would give animators more of a chance to win. This would work equally well with the website award, and if the Autodesk Inventor Award became regional too then it could also go along in this category. :]

Anyone against a Professional Award and a Student Award??

[opnickc]

wells heres my little bit and then ill probably shut up and let other people figure this one out....

any way you cut it i dont think that peer judging will every be....heres the dirty f word....fair...

my analogy here is this, and if im completely mistaken sombody slap me and tell me ive been ignorant.....

the chairmans award nor any of the design awards are not decided by peer judging...so why should the same be true for animation....and i know that the point of peer judging is to allow the animators to have more control....but still

so heres my comprimise idea....i know this is crazy....but 2 awards...let one be decided by peer judging....one rep from each time like this year....and one by judges....let people know what the professionals liked and what the students liked.....

---end idea-----



I second that (third, whatever). I think that, even though technical is only worth 30% of the score, students grade it as 90%. And when I say techincal, I don't mean the most difficult to make (aka biped and the like), but I mean which looks (and sounds) the best. Many students grade the other areas based on how well they are potrayed by the technical aspect, even if the animation didn't use complecated things like character studio. So yeah, having 2 awards (one student, one proff.) is a great idea. I hope FIRST will seriously consider this for next year.

On a side note, I must congratulate FIRST on doing a much better job with the animation judging than last year. All entries were on DVD and had no loss in quality, and the staff worked hard to make the judging as "fair" as possible. Hopefully next year will be even better.


[EDIT]
Anyone against a Professional Award and a Student Award??

I'm all for it.