Quote:
|
Originally Posted by MissInformation
Of course, there is always some wish that everything had gone smoothly and perfectly, but in the long run, I think overcoming obstacles together builds a stronger team. Sometimes lessons learned the hard way are the best lessons learned.
I do wish I could have changed the amount of stress and frustration everyone went through, especially our animation team (I'm not going to go into their nightmare). I like everyone to be happy, and it bothers me to see people upset, which is kind of hard to avoid during the crazy six weeks of build.
Heidi
|
Heidi won’t, but I will.
Sit back for a minute while we tell a bedtime story. I have dropped out a lot of the details, but it is still long. The following events are pieced together from the observations I was able to make while running back and forth between the computer lab and the robot lab. Some details may be a little off, but the heart of the story is there.
First, the disclaimer: our animation team is composed of about a dozen students this year, up from the three that did it last year. The two returning student animators from last year – Steven and Tristan – created their own Autodesk software course and taught their compatriots the software all on their own. They run the entire show. No adults provide any assistance – none of us even know the software.
As things came down to the final 96 hours, our animation team was all set to start rendering the final version of the animation they had been developing for the past six weeks. They had made special arrangements to use the computer lab, and had a 70-node render farm established and ready to go. They had the system all set up and ready to knock about the 900 frames of animation with an expected 3000 hours of computer time. A "start rendering" deadline was set for 7:00pm on Friday … which came and went.
First the render farm computers couldn’t find the textures embedded in the images. They worked this for several hours until we got thrown out of the school when it closed down. The first signs of "will we make it?" worries began to creep in. With special permission, they were back at 7:00 Saturday morning. Some of the renders started, but then the render farm couldn’t find the full scenes. The network wasn’t properly converting the aliases into the actual locations for the model and texture files. After a few screams of anguish, they started to manually convert the file names and restart the process on all 70 machines. Then as the models began to render again, the network choked once again on the file references that still were not converting properly. Slightly more screaming. Steven plugged his personal machine in to the network to serve the models to the rest of the render farm. Then they had to segment the entire job and manually command each node on the farm to start chewing through the first scene, six frames at a time. The only way to get everything done in time at this point was to make the anguishing decision to drop some items and some details, over which they had poured days of effort, from the final animation, and keep their fingers crossed that it would be done in time. So they started things up again, just in time to get thrown out of the school again late Saturday night.
Return at 7:00 Sunday morning to manually queue up more frames on the render farm. Everything seemed to be working, and like they might make it – until 11:00am. That is when the central server on the school network crashed. The main G Drive on the whole system goes down – hard. The I.T. tech support guy, who happens to be at the school, says it has overheated, and they need to call in their service contractor to get it up and running again. That is not a problem – except it is a holiday weekend and the rules do not allow him to place the call until it is passed through the county I.T. office on Tuesday morning. Oh yeah, and the G Drive is where all of the second scene happened to be stored. This time, MUCH more anguished screaming. Spend hours trying to find any fragments of the animation on any local machines. Then figure out how to reconstruct as many of the models and textures as possible from those versions – which were several days old. Find out how to drop some of the textures from the scene without destroying the models. Re-create others. Watch the clock, because we are getting thrown out of the building again at 6:00pm. Manually queue up all 70 machines to run as many individual frame renders as possible overnight.
Back at 6:30am Monday morning. A reasonable estimate of the rendering progress says they need another 500 hours of computer time. That is not including the frames that have to be re-rendered because somewhere in the network confusion a few hundred frames were rendered at low resolution and cannot be used. The animation team is on the edge at this point. The "will we make it?" worries morphed into "there is no flippin’ way we will finish, is there?" questions. All practical estimates of the work to be done result in a "no" answer to that question. But everyone jumps in and works to find a way out. Pull in a few more personal computers. Manually recover individual textures from old model files. Piece things together that were never designed to be combined. Dump everything onto Steven’s pseudo-server. Drop some details from the rendering, and one short scene, and then create new ones to preserve the quality of the final animation. Manually segment the rendering queue again. Collect the frames as rapidly as they can be finished. They have permission to be in the school until 9:00pm, due to a basketball game in the gym. But the game ends early, the school closes earlier than expected, with crucial frames unfinished. Aauugghh! Hunt down the custodians and beg and plead for an extra 30 minutes – "You can just start closing the building from the other end, can’t you? And get to the computer lab last?" It works! The partial render finishes, and they take what they have. Take them over to the house and dump the collected frames on one machine. Run the compositing software, test the codec output quality, drop in the audio tracks, and create the final version. Then keep trying until 4:30 in the morning to upload the final product to the Buzzsaw server to submit it – which takes at least three tries to get a valid version that can be downloaded properly.
Through it all, they never gave up. There were a few times that were close, but they refused to throw in the towel and admit defeat. With each new obstacle there were a few screams of anguish, but then they went ahead and pounded their heads against the problem until they found a way out. They improvised. They overcame. They persevered. And they succeeded. They proved they were ready to deal with the sort of real problems that a production graphics shop has to overcome. They behaved like professionals, and kept to their commitment to deliver a professional-quality, complete product without compromising their standards.
I cannot remember being more proud of a group of students with which I have been associated.
So what would I change about this season and their experiences? Not one single thing.
-dave