A solution for Animation Judging for 2005

[TKatsAniMentor]

Hello All,

I am one of the sponsors for the TechnoKats (45). We, myself and the other sponsors, have to say that the judging and overall enthusiam for the animation competition is at an all time low. Below are some comments and suggestions that we have discussed about the competition.

Regional judging still has problems. Not every regional has the same amount of teams attending; Therefore, teams at the least attended regionals have a much better chance of winning the award. After checking the release from Autodesk, we noticed that the NYC Regional has only 3 animations to be judged, while the Great Lakes Regional had 27 entries. Also, varying judging conditions and locations have also been a problem. Until these discrepancies are addressed, regional judging will not be "fair."

Another thing that needs to be addressed is the support and involvement of Autodesk with the animation competition. Autodesk does supply software to all the teams to create an animation. Does anyone use this software? We use AutoCad, 3DS Max, Adobe Premiere, Adobe PhotoShop, Macromedia Flash, Sound Forge, and Acid to do our animations. Only 1 piece of our software list is provided. I have heard a nasty rumor that Autodesk is going to force us to only use what is provided or the entry may be DQ'ed.

2002 and before Autodesk had a group of industry professionals that graded animations on Technical Execution, Content, and Creativity. These areas had their own awards as well as the overall and rookie awards. In 2003, Autodesk decided it would be best to eliminate the other honorable awards and give only the overall and rookie. I have to say being the Technical Excellence Honorable Mention Award winners in 2002 that we were upset about the change. With the judging leaving the hands of professionals and it being thrown at the students, all the current problems arose. First, the animation creation time was reduced by 1 month to coincide with the robot ship date. This caused students to choose either to work on the robot or the animation (in most cases) and forced the animators to lose more sleep due to the shortened deadline. Second, judging at last year's competitions was a total joke. Lastly, the entry that won the whole thing did not even win a regional competition and was mostly video, not animation.

Now that most of the problems are now addressed, let's talk solutions.

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

Anything else you would like to add to the solutions list is welcome. Please no more gripes. There has been plenty of threads of those already. This was created to try to brainstorm solutions to fix the current problems.

Thanks for reading this.

[Joe Matt]

While I was not on animation this year, I watched and somewhat helped (ok, not much) them. The main focus of the media team was the animation, so I know where some of the problems are.

Originally Posted by TKatsAniMentor

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

1.) I agree, but also disagree. I think that students should have an opinion too in the animation award. I think they should judge for the things they did in 2002, but then have an 'Overall score', that'll count 60% of a 100% animation score. The other 40% of the score will be done by ALL students, not just those who submitted animations.

2.) Yet again, I agree and disagree. I think that the animation should be given another month, but make it before the kickoff where you add the time. Release the theme and what you have to do, then allow for teams to use the winter break to create textures, ideas, and concepts. Then when the game comes out, lots of the work is done, and if teams want that year's robot in it, they can add it, without worrying about the other things they have been working on for a month. Ship along with the robot.

3.) Totally agree.

4.) Which answers the question, like what? A technical challenge, or a thematic challenge?

5.) At VCU we had one student sent to watch the video in a private room with a wide screen and all. I think this is good as it is.

6.) Agree again.

I don't think we'll see the animation as big as the actual competition, just like how the web site won't nor the business aspect of it. If FIRST wants to do something great, make the animation part totally separate from the robotics part. Think of it as an art contest and less of PR for your robot.

[Collin Fultz]

Originally Posted by TKatsAniMentor

2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?

I totally agree with your second point. It really inhales audibly to see some students not be able to build the robot or work on the animation because they are at the same time. However, I can see why FIRST does this, and understand %100. There are some teams who don't have 24/7/365 access to computer software. This technology is only available during the six weeks. Some teams are lucky enough to have virual any-time access (234 included) and I'm thankful for that, however, I don't think it is "fair" (I hesistate to use the "f" word) to teams that don't. FIRST has reasons for everything. I see and understand this one.

The third point is what I have a main problem with. As Andy Baker said, "Don't concentrate on the entry itself, concentrate on what you do for the entry." (or something like that, sorry if I screwed it up AB) He was talking about the Chairman's Award in this quote, but it applies to all of FIRST. The Kit of Parts does not come with a rubric of what will win this year's game, nor should it. The Chairman's Award info does not come with a rubric of what is judged and what isn't, because the point isn't to win the award but to be inspired and inspiring while doing it. Cyber Blue runs a total design review process. We don't do this simply to win awards, we do it because it is a part of some fields of engineering and even if you plan on being a bum, presentation and oratory skills are vital to sucess. When students walk into the Chairman's Award presentation, they have some idea of what they are being judged upon, but I have never seen the official rubric of how I am being judged, and I'm glad for that. If there isn't a rubric for "FIRST's most prestigious award" (that's in quotes as a quote, not as sarcasm) why should there be one for anything. Once we start creating animations/chairmans awards/robots/lives simply to satisfy a rubric...the creativity is lost, the passion is gone.

I love watching animations. About all I had to do with ours was to sit and say, "Man, that's really good. Good job Ryan/Chaz." I think animations and animators are some of the coolest things in FIRST, but that doesn't make it fair. I heard somebody from our team say, "Yah, Gunn always has a cool animation." They won Arizona's Animation Award this year. Does that make it any more <shudder>fair</shudder> than when it seemed like Hammond was invincible two years in a row? "Life isn't fair."-Dean Kamen There is no letter in FIRST for Awards. The only reason I believe "Recognition" (from an award standpoint) is in there is because "FIST" isn't that inviting. Do it for the "I". Be proud of your team and your robot and your chairman's and your mentors and your animation and and and. Don't do it for the award. Again, I'm not an animator by any aspect of the word...however this stems much deeper than animation.

/climbs down from soapbox

[TKatsAniMentor]

Originally Posted by collin234

The third point is what I have a main problem with. As Andy Baker said, "Don't concentrate on the entry itself, concentrate on what you do for the entry." (or something like that, sorry if I screwed it up AB) He was talking about the Chairman's Award in this quote, but it applies to all of FIRST. The Kit of Parts does not come with a rubric of what will win this year's game, nor should it. The Chairman's Award info does not come with a rubric of what is judged and what isn't, because the point isn't to win the award but to be inspired and inspiring while doing it. Cyber Blue runs a total design review process. We don't do this simply to win awards, we do it because it is a part of some fields of engineering and even if you plan on being a bum, presentation and oratory skills are vital to sucess. When students walk into the Chairman's Award presentation, they have some idea of what they are being judged upon, but I have never seen the official rubric of how I am being judged, and I'm glad for that. If there isn't a rubric for "FIRST's most prestigious award" (that's in quotes as a quote, not as sarcasm) why should there be one for anything. Once we start creating animations/chairmans awards/robots/lives simply to satisfy a rubric...the creativity is lost, the passion is gone.

I have to disagree with this. I am a educator and when I assign a assignment I require certain things. This does not limit my students' creativity, but helps them direct it. You said that there is no rubric with the kit of parts. I think we are talking about apples and oranges. I said what I said due to the fact that Autodesk has no consistent vision for the award. A rubric would convey their vision to all so everyone has a chance at the award.

To use the "f" word again, The whole competition will never be "fair" with the current structuring. Every team has different expertise, tools, and technology. At least with the animation, a little more direction would be nice other than just "create a commercial for FIRST."

[gsensel]

Originally Posted by TKatsAniMentor

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.

Totally agree, as much as the students do on an animation professional should look at it. This is so they know what truely will fly in the professional world. Many student unless they are activly immerssed in the world animation can not acuratly tell what is award worthy.

Originally Posted by TKatsAniMentor

2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.

Agree because it is a thrill when you see a robot that you built, in some way, competing.

Originally Posted by TKatsAniMentor

3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.

In schools even for little assignments a rubric is given to provide an idea, or possibly direction it is intended to go.

Originally Posted by TKatsAniMentor

5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.

Possibly a change for this could be a judgng that occurs before regionals, maybe later in the season so the students an do both animation and robot, is done by professionals.

Originally Posted by TKatsAniMentor

6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

Totally agree, if nothing else it recognizes more than just 2 teams in a season. Also it affirms what the judges like.

[ThetaDot]

The point of student judging is to bring us into to fore a bit more. I have confidence in the kids that did the animations to be able to judge decently enough.
I had no problem with the organisation of the judging; it went pretty smooth - at least at GLR.

Also I think part of the ambiguity in what the judges want has a purpose. We are free to do what we want and to show it how we want to. If a finely detailed rubric gets distributed to us, we are going to end up seeing the same animation over and over. Personally, I made what I wanted, without regard for what would get me a win.
I agree with what Collin234 said. To do some of the things you suggest means making it less of a challenge in my mind.

Also, I do not agree with separating the animation work time from the robot build time. The visualization is an optional component and so anybody pulled between the two should obviously drop the animation and go for the robot.

This is the first year our school had done FIRST, and I don't think anybody on our team has had any complaints. I know the animation competition isn't perfect, but I am not going to ask for perfection. I am content with how it exists today.

[reisser]

The professional judging seems to be consistenly inconsistent. There must be rubric so that the "professional" judges know what to look for. I quote the word professional because certain hints in the comments we received in the past suggest that some of the national judges are not professional animators. Regardless, a set of universal criteria is necessary to make a fair and consistent judgement about a work of art.

For the past three years I have read national judging comments that contradict themselves. Some judges harp on about the "lead in" statement while another judge requested more "sexy robots". One judge commended our use of depth-of-field when there was none at all. Little inconsistencies such as these are very hard to take when we all have worked so hard. And while the piece that won last year was compelling, I have to agree that it was disappointing to see a winner picked with so little demostration of 3d animation. Again, this clarifies the need for a judging rubrik. It is to keep the judges on-task, fair, and consistent. I struggle without a clear set of criteria at regional judging. I think any judge would benefit from knowing what to look for instead of assigning arbitratry number values for a vague set of categories.

The competition documents contain hazy instructions at best. Every year there is some misunderstanding about the "credits" version and where to put a black frame. Two years ago our score sheet reported that out entry arrived late. We Fedex'd it four days early and received deilvery confirmation. Some aspects of the competition are just blatantly disorganinzed.

I have been hoping that postivie changes for the animation competition would occur. It would be really rewarding to have some good come out of our criticisms. See everyone at nationals.

[Jonathan M.]

The thing I have the biggest problem with is the deadline date. I'm the Animation leader of the team, and we couldn't even do one because the entire animation team had other things to do that concerned the robot.

I hope Autodesk doesn't limit us only to what it provided. I really like to have my other software around to help me out. Not saying that we couldn't do it with just 3Ds, but it would suck.

So yeah.. again my biggest problem is the deadline date. Hope that will change.

[Salik Syed]

I serioursly think there are problems with student judges because ...well lets just make a scenario

Person 1: They judge the animations by comparing it to pixar....(mostly low scores)
Person 2: They compare the animations to an ideal student made animation (medium scores)
Person 3: They compare other animations to their own (which lets just say isn't good) (mostly high scores)
now if person1 judges 2 & 3 but they judge 1&2, 1&3 then obviously person 1 will win!!!
now this is REALLY unfair esp. at a small animation event with like 5-6 teams so that discrepencies can't sort themselvs out

[av11d]

Yes, scoring is VERY inconsistent. Last year, we received an atrocious score from the judges in the technical portion: A 1.1 out of 20 if I remember correctly. Our animation featured character animation, bipeds, and lip syncing. We were using some of the most advanced features of 3ds max! I feel we used all those features successfully, and so did other teams, because we won 2 (or 3?) regionals.

I also agree with Salik that a scenario aforementioned could easily happen. There is no standard to base the grading off of. 0 to 100 is huge. How are you supposed to know what a 50 is? A 50 could mean average, since it's half the max score. A 50 could mean atrocious if you think about in the sense that it's a 'failing' grade. So who knows!

[kcy88]

This is really great that us animators are addressing certain issues. I was disappointed at the fact that 103's animation wasn't played when they won the award at the NJ regional. If i recall correctly, last year, they showed the winning animation on the big screen, but this year... what happened?

[SteveC116]

The four times that I have viewed the awards ceremony (this year and last year), I never saw the animation played at the end. Normally it's just a powerpoint slide.

[Ryan M.]

I agree with everything you said. However, I can understand the motivation behind limiting teams to only their product. Maybe if teams weren't allowed to use anyother commericial products, that would work. One of their reasons for limiting what products can be used is that many teams can't afford to by Adobe Premier, Flash, etc. So, if you were only allowed to use the supplied software and any freeware/open source stuff you want, maybe that would be a good compromise.

[RoboMom]

Animations were shown on the big screen while waiting for the awards ceremony at Chesapeake on Sat. It was great to be able to see all the entries and this should be an important part of any regional.
Unfortunately, our really hard working animator, team of one, didn't get done by deadline. I give her a huge amount of credit though. She taught herself completely, and had to work using the one licensed copy on one really lousy school computer, never once complaining. I hope she will finish it to put in her portfolio.

[Ryan Dognaux]

Originally Posted by TKatsAniMentor

1. Autodesk needs to bring back the panel of professionals for judging. This meant that the judging was consistent because everyone saw every animation.
2. The animation competition needs to not be at the same time as the robot build. This gives the students an opportunity to work diligently on both.
3. We need a judging rubric. How are we supposed to know what it is that we are being judged on besides general categories?
4. We need a clear objective to achieve with the animation. This will give the animators a goal to shoot for (just like the game). This should not limit creativity.
5. Regional judging should be more organized if it is to continue. Maybe there could be a panel of team volunteers (students and mentors) to judge at regional competitions.
6. We need feedback. The honorable mention awards at least told the 3 teams recognized that they were close to what the judges were looking for.

1) I think Autodesk needs to contact Disney, Pixar, etc. and get a few real professional Animators out and about at the competitions. It's like this - many students are inspired when they meet and talk to Engineers, but if we (animators) never meet Animators, then how are we supposed to get excited and be inspired?? If FIRST and Autodesk could get professionals to make the judging professional, then I'd be all for having them doing the judging. Otherwise, keep it the way it is.

2) I agree with you to an extent. I think that Animation should've been due at the same time as the Autodesk Inventor Award.

3) I think all teams should receive the scorecard ahead of time since it has clear definitions for what you will be judged for (Concept, Creativity, Technical Execution).

4) I don't even know what to think here. I think it's amazing what all of us put out each year, so it's probably a good thing to leave it the way it is.

5) Again, I think a bigger presence from Autodesk would be nice. It's almost like the few people or the person that attends the Regional is invisible.

6) I agree with you totally here. Honestly, I think Autodesk does a poor job with relaying things to teams and the animators. Autodesk has a voice, but it needs a megaphone or two to reach us..... Is it just me, or does Autodesk seem just very disconnected in this whole process? Don't get me wrong, I'm very grateful for all they do, but it just seems like we don't hear much from them.

My 2 Cents...