I feel that I have to share this. I’m not sure how many teams here were at Richmond regional, but for those of you who hadn’t been there – here is how it went.
Our animation group arrives on Friday, full of hopes, and receives a voting ballot for the animation judging. Soon enough we are able to find the spot where animations are played, and guess what? It is located in the pits area, the noise level is above anything that the small TV placed there can produce, and the best part of all – you HAVE TO WEAR goggles in there! The feeling was that they were just hiding away the whole animation competition.
But that was only the beginning. We look through the loop of animations played and see that our animation is COMPLETELY screwed up by grainy artifacts, flashing white screens and color shifts. The whole thing is painful to watch. It is 8:30, and we start running around the competition area trying to find someone to talk to. Turns out – there is no one there right now who knows about animation at all, “but if you wait till 10:00 someone will show up”. We nervously wait till 10:00… nobody there… 10:15… still nobody… 10:25… finally, a lady comes saying that she is going to run the judging, but she is not actually an Autodesk representative, - just a volunteer helping out (the real representative had said that he couldn’t come so they quickly needed to find someone else).
We explain our situation, and ask if we could show our REAL animation on a laptop (luckily I’ve brought a CD with our animations on it). She agrees, and we spend another 30 minutes running around to find a laptop that could play our animation. (640x480 high quality mpeg is not something our control group’s laptop could run). Finally we are set, and there are some team judges starting to come to the table.
Judging. No one knows when the real judging began. Nor even the fact that there was animation judging was ever announced. There were a TV and a laptop sitting through 10:40 to 2:30 and teams just randomly came and judged. The TV was so small, that only one team could judge at a time, with a small line waiting for their turn. And yes, we had to sit by the laptop, embarrassingly explain to the teams that we are not crazy, and we did not originally have the seizure-inducing flashes on our animation, and would they please watch our animation on the laptop, and not on the TV? Most did… I know of at least 2 that didn’t…
Now for the scoring itself. Each team was given a ballot with team numbers, and rubrics listed for each: “Content”, “Creativity”, “Technical Execution”. So you had to fill out the numerical scores for each animation, say “30/40” content, “23/40” creativity, “15/20” execution. There are so many wrongs with this system, that I’ll need a list to mention them.
- Can someone think of a more subjective scoring system than this? I can’t. Every single team had different ideas of what the rubrics really were, and what appropriate scoring ranges would be.
- The final score for each team were AVERAGED – you don’t want your team to lose? Just give low scores to your competition!
- There are 25 animations – judging all of them takes 40+ minutes. Your scoring “scale” will inevitably shift during the process, which means that first 5 teams played are judged on a scale completely different than the last 5.
- Again, looking through all animations takes at least half hour. With the pressure from the teams waiting in line, it is next to impossible to spend appropriate time for judging each animation. Many teams just rushed through the scoring, not really thinking about the points.
- With all of the numerical scores that should have been added, averaged, etc. I can bet there were calculating mistakes during the scoring, they probably didn’t make much difference in the end, but the fact of their presence is still pretty sad.
The system was horrible without doubts. Implementation was even worse. Soon enough we found that there were teams whose animation was on the tape, but whose numbers were not on scoring sheets. There were opposites – they were on scoring sheets, but animation was missing on the tape. There were even worse cases, when the animation was on the tape, but title screen was cut off, making it impossible to identify and score it. Most of the teams who got screwed up like this were not even scored, automatically loosing. And some of them were pretty good too.
Continuing ranting about the tape. Artifacts – ours was not the only one covered with flashes, but by far the most artifacted on the tape. There were others with minor flashing (which nevertheless killed any possible good impression from viewing the uncorrupted parts) and graining. The best artifact I’ve seen – the “Low Battery” message that got recorded over team 21’s animation (actually I’m not sure about this one – if it was original part of animation, I take this back, but it really didn’t look like it should have been there. We’ll need someone from team 21 to clarify that) But that’s not all – the order in which animations were recorded was pretty “interesting” too. I haven’t noticed any discernable logic behind it – it was random at best. And there were duplicates on the tape! Some of the animations were recorded twice per loop! Why? Only Autodesk knows…
The last thing – could someone please, please explain me why did Autodesk divide the whole thing in conferences?? All of the conference animations get judged at regional, but only the teams who actually submitted to this regional can win! What do they do with the scores for the rest of the teams? Throw them out? Or what?
Finally, the award ceremony. For the first time we actually hear something said out loudly about the animation award. We nervously wait, still with some hopes of winning – we know that at least 2 out of about 6 teams that judged had seen the wrong animation, but we still have hopes. No luck on this one, winner being team 343.We know, that now its too late to argue about anything, but we still want to see if we are one of the two runner-ups that get to go to national championship for the animations. According to the rules: “The scores will be available immediately after the award ceremony”. Yeah, right – only there wasn’t anyone there who knew anything about the scores, and the most we heard is that Autodesk still haven’t decided if they want to release them right now.
When we arrived to competitions on Friday we had really high hopes for winning. And if there were no artifacts (which were definitely Autodesk’s fault – we have submitted our entry in Intel Indeo which fits perfectly under the rules, and we tested it for compatibility on numerous machines with different operating systems), it would have probably been so. And many of the teams that had actually seen the uncorrupted version had agreed with that. Our visual quality was nullified by Autodesk’s incompetence in transferring the video to VHS, and our sound track was thrown away by the noises in the pit area. Even if we had some chance with presenting our real animation on laptop – that was destroyed too, by the utter disorganization of the whole animation competition and the judging in particular.
But its not just about the scoring or judging. What I find the most sad is the whole attitude FIRST has toward the animation award. They didn’t bother to take time and write rules that make sense, find a decent place for judging, inform the public of even the presence of such award, - they didn’t even bother to look at the tape they prepared for the judging! I’m sure that if ANY official had seen it, it would not make its way to the regional.
In case any of the FIRST officials are reading this – here is a list of my suggestions for regional animation awards:
- The animations should not be played on a small TV in the farthest corner of the pits. Teams worked hard on their entries and it would be fair to recognize their effort – set up an LCD projector so that everyone can see it!
- The judging should take place in a separate, quiet, room, where the animations should be played off an LCD projector from a computer – not a VHS tape – this will remove 9/10 of the visual problems.
- The time of the judging should be announced so that everyone can hear it! Why do volunteers need to run through the pits asking if their team representatives had already judged?
- I know it is late to change the scoring system, but still few things can be made to make it at least somewhat more objective. Before judging, teams should re-read the rules, to see what rubrics are composed of. Also, more specific guidelines should be given for the points system – for example: 20/40 – average, 30/40 – very good, 35/40 – exceptional and so on.
- The scores should be posted somewhere immediately after the awards ceremony. And not only posted, but announced where they are.
Now that we are screwed, I don’t know where our entry will end up. Anyways, I wish luck to all of the teams on the upcoming regionals, and that FIRST would do something so that our story will not repeat.
Our regionals competition was won by Gunn robotics (team 192), who had a truely remarkable movie, by the way, but they didn’t come to our regional. At this point i am seriously considering whether or not to enter a movie next year. At least I can hope that the Silicon Valley regional will be better.