Is Peer Voting the Way to Go?

[Carl C]

I would like to make this clear up front: I'm not trying to put down any of the winning animations. You guys did amazing! And Autodesk does a great job running this competition.

I'm just not sure that peer voting is the best way to do the competition, mainly because I fear that it doesn't give the "little guys" a chance.

It's not as if every team already has an account. In fact, most of the teams that I talked to didn't even know that they could vote. I felt like there was no initiative to have every team vote, and thus it was up to us (the teams that entered) to get people voting.

But here's the problem with that: if I ask you to vote in a contest that I entered that you don't particularly care about, who are you going to vote for? Me, of course. It's the courteous thing to do. I asked you and it would be awkward to tell me you voted for someone else. Now, imagine if we did this for the chairman's award!

From my perspective, it seems that the way the regional animation award is set up, it gives an advantage to teams with more connections.

I think that the best animation should win, not the animation with the best campaigning.

Anyways, I'm going to get off my soapbox and I would like to know what you guys think. Do you see any advantages to the current system? I would love to learn, and I'm sure that I'm missing something. I do not wish to offend anyone, and I hope this post was both gracious and professional.

[Sunshine]

I totally agree.

Peer voting is not the way to do it. At least they have experts doing the national level.

[Katie_UPS]

I'm not in animation, so this is completely outsider perspective but:

I don't think peer voting is that bad. It lets animation teams see all the animations. They can see where they should improve, what they did well... If I'm remembering correctly, our animators went to a forum-type meeting to vote. This allows your animators to meet other animators, network, ask questions, etc.

I've only been around for four years, but I don't remember any "animation campaining" at any of the events I attended... I do believe that 90% of the time, the best animation wins.

...but like I said, I'm not an animator.

[Carl C]

The way the process works is that we turn in our animation, and a few days later peer voting opens up online. Each team only gets one vote, so there is individual voting. And it's online, so there is no inter-team animation meeting.

By "campaigning," I mean that one team makes sure another team votes, since that is the only way they find out about the voting.

[Cyberphil]

I completely agree. It seems very unfair to any team. I barley knew we could have oher people vote on the award, let alone tell other people about it. I was confused as to why autodesk did it this way this year, and hopefully they change it for next year.

[Koko Ed]

I've never seen teams try to influence a vote while I know of a FIRST team winning a technical award for a part of their robot they didn't even use so the judging process is just as flawed.

[Navid Shafa]

Originally Posted by Cyberphil

I completely agree. It seems very unfair to any team. I barley knew we could have oher people vote on the award, let alone tell other people about it. I was confused as to why autodesk did it this way this year, and hopefully they change it for next year.

It's been this way for a while now. Even the Autodesk Inventor Excellence in Design award was a voting contest back in 2008 (To determine the top 6 to be judged in Atlanta).

I understand that the amount of FIRST liasons which Autodesk has is relatively small. Seriously, it's only about a handful of people who end up reviewing the awards. They have enough to deal with...

I am not saying that it is the best or the right way to do it, but it's the most currently feasible way, with the resources they have. From what I've seen over the years, on the regional level, it doesn't matter anyways. Often there aren't that many teams at a single regional who apply and the winner is almost always very deserving and is usually a great call made by the viewers. I'm glad that there isn't a voting process at World Champs, because I agree, I think this process would be highly tainted if so.

I would love to see the Autodesk Excellence in Design Award for 3D Modeling some day become big enough that it is judged on the regional level along with animation. It's cool to see the other team's submissions at world champs and online, once released. However, I wish I could see more at the regional level. Once they start assesing it at the regional level, I think they will be shocked to see the amount of participants increase. This is not only good for Autodesk being a main sponsor, but for the teams as well, who are able to take a lot from it.

Sure I might be biased, being an Autodesk junkie on our team, but I wish that the message was spread wider about the voting and the competitions themselves. You think that as a world-wide sponsor, FIRST would be sure that they announced the animation and design entry process at every regional. Even just as a slide on the board reminding people where to vote...

[scrubsfan1793]

I personally do not have a problem with peer judging at the regional level. However, I do not like the way it has been done for the last 2 years. This is my fourth year animating, and I thought that the peer judging was nearly perfect the way it was done my first 2 years. You get a certain number of points for each criteria: Modeling, Technical Execution, Concept, etc. For example, my team won our second regional because, while the technical execution was about what you'd expect from a second year team, we had a great concept. And I'm not just saying thats the way to do it because we won with it, I think that's a legitimately fair way to do it.

On the other hand, nationals should be judged by professionals. That's an easy one for me

[Katie_UPS]

Originally Posted by Carl C

The way the process works is that we turn in our animation, and a few days later peer voting opens up online. Each team only gets one vote, so there is individual voting. And it's online, so there is no inter-team animation meeting.

By "campaigning," I mean that one team makes sure another team votes, since that is the only way they find out about the voting.

I apologize then. My freshman year, our animators went to an animation shindig at the Midwest regional and I confused that for voting.

I retract my statement.

[robertsun]

I really do agree.
The problem with the peer voting is there is so little people who actually vote. Sometimes, just the animation teams themselves. If we were to make this into an actual system, autodesk or first would have to notify every single team and get them to vote, or else something bad will happen. That way, there will be no bias and greater accuracy will follow.

[jgoldfin]

Interesting feedback - thank you.

i'd like to hear more about the peer voting process, as this was a very big part of the award process this year, and it took a lot of time and resources to set up and execute. We did try and communicate that all teams could vote, and also we set up the peer review pages, but i don't know how many people used them.

Peer review site - not the voting site:
http://autodesk-firstanimation.skild.com/PublicVote.jsp

with the number of regionals and the number of entries increasing, i am not sure we could get professionals to judge the regionals, but we will be asking you for your feedback on the process and do our best to improve it.

Thank you -

[Galum]

I totally agree.

Peer voting has other issues, especially in regionals where animation crews are hard to find (such as Israel). For example, last year the only team to vote was our own! So we lost the regional because there was only one vote, that's hardly fair in my opinion (we learned that from the Regional Directors).

It's really hard on the kids, the worked really hard completely on their own (we couldn't find anyone with knowledge of 3DMax near us) and felt that they were robbed of their fair chance.

[Kims Robot]

Being that the Website award is essentially the same style (a pre-judged competition), could the Autodesk awards be done the same?

My understanding of the Website Award is that each regional recruits volunteers (presumably technical mentors who perhaps know something about websites) to do the evaluation ahead of time. The evaluators have a criteria sheet that they judge each site against and each site gets a certain number of points.

Could the Animation award be done the same?

The Peer review is a bit nice in that everyone that wants to gets to see all of the entries, however, as stated, it can be a bit of a "recruiting teams to judge" type thing. I agree that ultimately the deserving teams tend to win anyway, but I can see the merits of having a slightly more subjective judging process.

I thought a couple of years back (maybe 5 or 6 now...) they had a voting kiosk at the regional.. possibly at champs? Though I kind of left that up to the lead animation student & mentor, so I don't recall the details or if it was just a display of all of the animations. Having it judged by peers at the regional *might* be a little more even as it is a more direct "VOTE HERE" type of thing, and could even be done like inspections where they have a board checking off the teams that have already voted. We actually do peer voting for the Off Season Ruckus event (though the committee breaks ties & distributes the awards), and its a fun way for teams to help decide who they think deserves the awards, while taking off some of the stress of having to recruit/employ judges. We give every team one sheet, and our pit admin makes sure every team turns in their vote.

Though I think its very cool that Autodesk is listening... even here on CD!
*crossing my fingers that NYS teams can compete next year*