View Full Version : How do you feel about how the Autodesk award winners were decided?
R Stephenson
16-04-2007, 16:33
How do you feel about how the Autodesk award winners were deiced?
DO you think that after the top 5 that it should be 100% based on vote?
How do you think the decision should be made?
please include:
Does you team take part in the Autodesk award?
I think it is completely up to Autodesk. Not because it's their award or that they give so much money to FIRST (which is absolutely amazing by the way). But how Autodesk decides to judge awards will inenvertantly change the products of the award. This affects nobody but the company themselves, and by allowing students to vote for the Inventor winner, we are changing our audience from engineers and designers we have never met, to our peers and fellow Firsters. We are now not trying to impress engineers but students and common people who are coincedentally the audience for Autodesk. In essence what I'm saying is, this may be a positive change for both Autodesk and the award's growth.
P.S. I apologize for mispellings I just got out of a Physics Graded Review.
I'm not sure. I think there are positives and negatives.
+ The good thing is that more people are likely to be interested and want to submit and participate in voting if they have that opportunity. It also allows teams to see what the other teams have submitted and first year teams can learn by observance. Autodesk still chooses the top five, and then leaves it up to the rest of us. The engineers are still involved and so are we.
-The only downside I can think of at the moment, would be Autodesk's involvement. It might seem to some outsiders that they just sit back and wait for all the votes to roll in, announce the winner, and they're done. We know it isn't that way. I also think it might seem unfair to some teams, or individuals. It just depends on personal preference.
Our team does take part in the award, and I personally have no preference; either way works for me.:)
Evenstar_342
16-04-2007, 18:45
First of all, the Design Subteam of Team 342 would like to congratulate Team 103 for winning the Autodesk Inventor Award.
I'll take the risk of speaking on behalf of the subteam. After much discussion through out the build season and after the top 5 were announced by students and mentor, we are very disappointed that Autodesk has chosen to have the award decided by student voting. With all due respect to our fellow FIRST students, we feel that even other design students (and overselves) could not possibly try and determine what, technically, is the best entry. We understand that to the best of our knowledge, the Autodesk Inventor award is to honor the best technical entry. Therefore, we are unsure how we, as students without the team's files or methodology, could decide which entry is technically, the superior (although we are biased, of course, and would like to thank all that voted for our entry).
We would like to see two things. Firstly, we feel that returning to the original judging by Autodesk judges, which was fair and impartial, is the best option. Unless Autodesk is trying to encourage teams to market their "product", which is a valid challenge and if so, this reason needs to be publicized; the voting process encourages students to create the best looking product and not necessarily the best technical product. We feel that deemphasizing the technical aspects of the award is sending a poor message to future designers. If Autodesk could clarify what they meant by changing the voting process (other than "we want to get the FIRST community more involved" which they did, in my opinion, poorly; almost no one knew about the contest or how to vote that I spoke to), it would be greatly appreciated.
Secondly, we would like to see the vote totals released for the overall contest (not per team, just how many people total voted). Therefore, we could see if our efforts helped get the word out about the change or whether we, as design teams, have a lot more work to do.
I don't mean to be beating up on Autodesk, these are just the general thoughts that are floating around our subteam as we review the past season. If anyone here is from Autodesk, we'd greatly appreciate any insight, especially if we've missed any sort of announcement about the award change. We also would love to see more teams submitting or at least looking at teams' Inventor work and if this inspires a student to delve more into Autodesk, than obviously that is a great development!Good luck to everyone next year!
Robert Thacker-
16-04-2007, 20:04
We have had exactly the same thoughts roaming around in our heads at the beginning of the season. And I still think that this peer voting can cause technicalities. We are really enthused about winning but we wonder about how teams vote. Everyone gets one vote, I think it may be better if a team as a whole gets one vote. Otherwise, what stops a team from creating dummy sites of people (say parents, teachers, or even people that are not part of FIRST) that cast votes for their team. I am not accusing any teams of doing that, but it is a problem that may come up one year. I think the way the award was before was better, personally.
We had a thread earlier in the year where we discussed all of it. Not sure where it is though.
(and thank you so much team 342!)
Peer voting is a nice thought, but... the way it was structured had two weaknesses.
First, teams were allowed to vote for their own entry- bad idea. That is the best way to invite bias into the competition. Also, all individual student members are eligible to vote. This sounds cool at first, but it sort of gives larger teams an unfair advantage doesn't it? Robert, you made some other legitimate points on this as well.
I love the way that the regional AVA was set up. Rather than determining a winner by vote, all entries were scored (a much more subjective way of identifying outstanding work). And participating teams were allowed one evaluation per entry... excluding their own. You still get peer input, but it's straight from your competitors. Now there's some healthy competition!
All in all, the result of this years competition was amazing! Great job to all Autodesk design teams, and especially to 1625 and 103 for their winning entries :D
Daniel_H
16-04-2007, 22:07
I saw some people voting at the Autodesk booth in Atlanta and it seemed too fast to be an accordingly judging. This is the main problem in my opinion. The old way with judges checking every aspect seems better.
Anyway, congratulations team 103, I sincerely think you deserved it.
=Martin=Taylor=
16-04-2007, 22:40
I'm not a sore loser... Not at all...
However, I am not particularly interested in entering the Auto Desk Contest next year.
I understand the importance of beautifying your entry and including every detail - its impressive and demonstrates mastery of the program.
But it isn't practical! you don't need to model all the chains and wires! :ahh:
Teams that go the extra mile to include those details deserve the championship trophy, but teams that use CAD simply to design and demonstrate robot functions need to be encouraged too!
I would like to see regional awards for CAD that would give the little guys a chance.
I would like to see regional awards for CAD that would give the little guys a chance.
The problem with this idea is that there are some regionals that have maybe one or two teams with entries, and you can't just give away awards like that.
BuddyB309
17-04-2007, 14:17
the voteing had some advantages as well as disadvantages.
-I like how autodesk was trying to get more students involved. But many students didn't know that they were suppose to vote for the autodesk awards. They didnt even know that there was a firstbase site. I like how the judges screen out top five and leave the final desicion up to the students. It makes the contest more fair then it being juged by some people we have never met before.
-After the top five were announced, the voteing seemed like a giant campain "VOTE FOR ME!!" The structure of first base was very frusterating. First a member of the team had to fill out there personal info and then send an email to there account so they could activate it. Then they had to wait to recive an email from their team cordinator so they can get the team password so their account can be fully oporational. If this is happening all at the autodesk booth, there is no way that they are going to get the team password in time. My team was pulling people from the pits and asking them to vote. Most of the time the student never heard of firstbase. Everything seemed like a mad race to get votes at the end.
Another thing i would like to point out is that the students who have never messed with 3ds max before dont know what to look for when voteting. I didnt know what to look for when voteting for the inventor award untill some one came and said "Oh, just vote for this team." and i did. I felt like i was doing the same for my team. People didnt know what they were looking at untill i came along and said "Oh just vote for 1625." I dont know, It something didnt seem right.
Theflexmaster
17-04-2007, 14:42
Personally I'm still a bit bitter about not being picked in the Top 5 for Animation, but whatcha going to do :( . In the end, I don't think having voting by peers is professional by any means. With voting, the most colors or special effects wins. Most don't take into account the difficulty of projects among other things. So I'm saying don't have voting unless by judges. Voting is always biased no matter what.
Skute327
17-04-2007, 17:17
I personally don't agree with the voting system at all, and I wrote the following to the autodesk feedback section
I Highly dislike the voting system, it is very unfair. it turns the award into a popularity contest. a larger team has a huge advantage, and most teams have not even heard of this site , and don't have accounts here, it is difficult to create an account, I am student team leader and it took me 2 days to find out my "team password" so I could create an account to vote for my own team. many people don't understand the idea of the award, and aren't sure who to vote for, if voting system is kept, it should 1 be more publicized, 2 be easy to sign up for, and 3 be one vote per team.
after reading again, and posting here i realize it was a little harsh and not very GP, the way i worded it, but i do feel this way.
I don't think having voting by peers is professional by any means. With voting, the most colors or special effects wins. Most don't take into account the difficulty of projects among other things.
I completely agree with this, and I couldn't have worded it better!
Folks,
First off, congratulations to _all_ the teams for their hard work and a great season. It is encouraging to see the quality of both the designs and teamwork improve year to year.
We appreciate the comments and want you to know we are listening. This was a pilot year in a peer voting method, and of course we want to improve upon it in the coming years. Keep the comments coming, especially while they are fresh in your mind, and know that they can have an impact on how we structure the award in the future.
Kind regards,
Pete
Ben Martin
18-04-2007, 16:01
What if, on Autodesk FIRSTbase, the main contact of each team designated one student to score the Autodesk Inventor award? This student could score each of the top five entries like Autodesk scores them. The downside here, though, is that you still have the chance that the student picks the entry with the most outstanding graphics.
In addition, we could include the opinion of Autodesk as well. Autodesk could take their score and add it to the average scores of all the teams. The team with the highest combined score would be declared the winner.
In this way, Autodesk gets 50% of the input on the winner, and FIRST Robotics students get the other 50%.
Anyway, it's just an idea. ;)
R Stephenson
18-04-2007, 19:53
... i didn't like how it was 100% voting, because it ended up being a popularity contest. and not about the work but i think there still should be a vote, keeping the students involved.
Autodesk gets 50% of the input on the winner, and FIRST Robotics students get the other 50%.
This is exactly what i was thinking but with more Autodesk involvement also like the idea of one vote per team it gets rid of big teams having an advantage, the registering process has to be much easier it took a while to register which probably turned people away from voting.
I would like to see regional awards for CAD that would give the little guys a chance.
I like the idea, don't really know how to do it, but maybe something on a smaller scale, like having a best in country .. and maybe because of the number of teams in the US have a best in the region such as "best in New England"
Daniel Brim
18-04-2007, 20:24
I don't like student voting.
I don't like an all-Autodesk vote, either.
Perhaps it is time for Autodesk to kick voting up a notch. I want the feedback (and votes) from people who use the respective software (Inventor and 3DSmax) for their job/career. I want to know how people who are closer to a real design process (read: engineers) to decide which design process/CAD integration is the best.
In short, I think that the awards should be decided by Engineers (Inventor award) and Animators who use the software in question to judge the entries. Thoughts?
Evenstar_342
18-04-2007, 20:49
What if, on Autodesk FIRSTbase, the main contact of each team designated one student to score the Autodesk Inventor award? This student could score each of the top five entries like Autodesk scores them. The downside here, though, is that you still have the chance that the student picks the entry with the most outstanding graphics.
In addition, we could include the opinion of Autodesk as well. Autodesk could take their score and add it to the average scores of all the teams. The team with the highest combined score would be declared the winner.
In this way, Autodesk gets 50% of the input on the winner, and FIRST Robotics students get the other 50%.
Anyway, it's just an idea. ;)
BMartin,
That's a really cool idea! I would think that the Autodesk portion of the award score would outweigh the guy who picks the entry with the "outstanding graphics". Out of curiosty, since we seem to have an Autodesk person watching this thread, can anyone explain why the award judging was changed? It just caught everyone on my team, at least, by total surprise and there are so many rumors and theories floating around, I'm sure Autodesk would like to get the official version out there.:confused:
Padre_Vez
19-04-2007, 14:56
At first I thought the voting would be a poularity contest. So I thought our team would not come in first. However maybe your right and just alot of people like our team.
Anyway thanks to all the teams that voted for us at nationals, and if anyone has any questions I would love to help.
Lil' Lavery
19-04-2007, 15:07
I'm in favor of the split-scoring system. Autodesk first selects 5/10 finalists for each award. The Autodesk judges each get an input, and the voters/peer scorers get a certain input. The weighting of each input is evaluated by Autodesk.
The criteria should be technical in nature, but beautiful graphics, marketing, and detail should play a role as well. In reality, you don't only have to create your design, you have to market it. Your design may be perfect, but if a/your company doesn't want to make it, it's worthless.
Robert Thacker-
19-04-2007, 22:19
I think that's a good idea that Daniel Brim suggested. The problem is that the top guys in the nation who use the program professionally may not care about FIRST and may not wnat to find the time to take place in judging. It's a harsh reality that Autodesk may have to face. I have no idea about animation, but I know Inventor is usually judged by very well-qualified people. Although knowing what kind of people who judge the awards would be cool. As in what THEY use the program for and perhaps some examples of their CAD work. (I'm curious if I could possibly top their work too - lol)
Skute327
19-04-2007, 22:52
I'm in favor of the split-scoring system. Autodesk first selects 5/10 finalists for each award. The Autodesk judges each get an input, and the voters/peer scorers get a certain input. The weighting of each input is evaluated by Autodesk.
The criteria should be technical in nature, but beautiful graphics, marketing, and detail should play a role as well. In reality, you don't only have to create your design, you have to market it. Your design may be perfect, but if a/your company doesn't want to make it, it's worthless.
the idea of the award is not to try to sell anything. inventor is a program to help design a product not sell it. Agreed, detail and ''pritty colors" are important for the business/ marketing end but in my opinion aren't nessissary to the drawing. i believe the most accurate representation of the team's robot should "win"
the idea of the award is not to try to sell anything. inventor is a program to help design a product not sell it. Agreed, detail and ''pritty colors" are important for the business/ marketing end but in my opinion aren't nessissary to the drawing. i believe the most accurate representation of the team's robot should "win"I don't mean to pick on you, but in my experience there are often times when designs are used to sell a project to a customer. In these cases the presentation is very important, and how well the model conveys the final appearance may be as important as how well it explains basic functions. I see the Inventor Award as a way students can learn these skills and marketing techniques.
A recent case in point: my department provides engineering services for the company's scientific groups and we recently completed the development of a high speed reagent dispenser for the biologists. We worked with the biologists to determine the best device to fill their needs, which included surveying and testing commercial instruments as well as proposing our own proprietary device. That meant that we had to "sell" our device as a design concept vs. actual instruments they could bring in the lab and test.
We were working with an external engineering company who provided basic designs and layouts in AutoCAD 2D format. The biologists found it difficult to grasp the functional concepts that way, so I created Inventor models from the ACAD files, including an animation of the product transport system and a concept model of the overall instrument package. As soon as the biology department leader saw those he went from a tentative and wary "customer" to an enthusiastic supporter of the project. Being able to visualize the instrument in his lab was a very important factor in getting his approval.
By the way, that external engineering company has since begun using Inventor and the project was very successful - providing our biologists with the best solution for their needs and giving the company a competitive advantage in what's know as "high throughput screening".
Evenstar_342
20-04-2007, 10:19
I definetly agree that how something looks or is packaged can help make a sucessful engineering product. I think that the controversy (at least in my team), is that the Inventor award is supposed to be for technical excellence. If Autodesk wants to include that a portion of each entry's score is based upon how well they've packaged their design (which I think is a great idea, as the world gets more globalised and more competative, we all should learn some marketing skills), that's fine. I think what we're having a problem with is that we're not sure whether we should be marketing our design, or even what criteria we're truly being judged on during peer voting, since everyone has a different set of criteria.
Daniel_H
20-04-2007, 11:25
Judging Criteria
Judges will review all entries. Five finalists will be selected to advance to peer voting.
In scoring your entry, judges will address specific criteria:
Robot Design 50 points
Technical Expertise 30 points
Presentation of final design 20 points
TOTAL POINTS: 100 points
This was taken from FIRSTbase website, I believe that's the way Inventor Award should be judged by other people too, not only for the judging of the top 5.
The only way to judge properly these 3 areas is looking carefully at the Inventor file and reading all the website that the team developed, but to do that you have to know the software very well.
There is no way to make a voting system that is non biased. Even if you were to make one person per team give a score for every other team, what would stop a team from simply getting every sister team to give them hundreds and everybody else zero's. I'm not saying that cheating would be rampant but it is peoples natural instinct to favor themselves or those close to them.
Although you are all looking for a perfect system, there is none. As long as we know how the award will be judged, teams will always be able to adapt their award to fit the criteria.
Also you do want an aesthetically pleasing design. I know for a fact that the Air Force uses programs like Inventor to prepare their Alpha prototyping. This is how you sell a product, because you cannot simply make a 2 million dollar aircraft to show off.
Overall it shouldn't matter how the award is decided, it will never be 100% fair, but this is still by far an award with one of the most spelled out definitions every year, you know exactly what to look for in a submission.
Robert Thacker-
20-04-2007, 20:27
There is no way to make a voting system that is non biased. Even if you were to make one person per team give a score for every other team, what would stop a team from simply getting every sister team to give them hundreds and everybody else zero's. I'm not saying that cheating would be rampant but it is peoples natural instinct to favor themselves or those close to them.
Although you are all looking for a perfect system, there is none. As long as we know how the award will be judged, teams will always be able to adapt their award to fit the criteria.
Also you do want an aesthetically pleasing design. I know for a fact that the Air Force uses programs like Inventor to prepare their Alpha prototyping. This is how you sell a product, because you cannot simply make a 2 million dollar aircraft to show off.
Overall it shouldn't matter how the award is decided, it will never be 100% fair, but this is still by far an award with one of the most spelled out definitions every year, you know exactly what to look for in a submission.
Couldn't have said it better myself.
Concerning AVA,
I'm a little tired of the voting system changing every year. Our team has submitted into the AVA for three years now, and each year we've had to submit or judge the awards differently. Last year, Autodesk turned to online voting, which ended up as a disaster in the regionals we attended, with teams being called to vote at the event and many not voting at all. This year, we had to submit the animation via firstbase, which ended in another batch of problems. Our school server blocks all upload sites, and when we tried uploading from different locations, the upload would frequently fail.
Personally, I believe the best way to decide would be a combination of last year and '05. Teams would only be able to submit to one regional, the teams would vote on these animations during the competition, the winners of the regionals, along with a handful of other team animations would then move on to the finals, which will be judged solely by Autodesk.
The reason why I feel this way is because I think that online voting just doesn't work. Most people and teams don't know that Firstbase exists and when they try to register, they have to go through a long and arduous process to join and vote.
I wish the adults were allowed to vote too.
Our opionions don't matter?
I wish the adults were allowed to vote too.
Our opionions don't matter?
Not when you can't spell. :p
Not when you can't spell. :p
It's not my fault CD dumped the spel chek!:p
JaneYoung
22-04-2007, 23:01
It's not my fault CD dumped the spel chek!:p
Welcome home, Ed. :)
Robert Thacker-
23-04-2007, 07:41
One thing I have a problem with is Autodesk not updatting their website. Their "FIRSTbase" still says that there are only finalists and it doesn't say who won. Even in the team page, it still says "Submit your entry today."
If a company pushes a new aspect of the award, they should step behind what new changes they decide to do.
I like the idea of adults being able to vote (because then I would be able to), but then teams could just contact all of their alumni and get them to do it.
Robert Thacker-
23-04-2007, 15:02
Personally, I think the lesser of two evils would be each team gets one vote.
teammember_342
23-04-2007, 21:30
Whoever started this thread should first write "decided" correctly!!!
Great job for another Autodesk award win Team 107 three times in a row!.:D
Anyway, this year was my first time on an FRC team and it was very interesting. I joined the Subteam of design on Team 342 because I am interested in Autodesk. I heard the process of last year was better so why change it. If this was alright as it was last year then let the Autodesk judges pick the winner.
Great job for another Autodesk award win Team 107 three times in a row!.:D
Um, team 103 won, not 107.
Thanks for the support. Good luck next year 347:).
Evenstar_342
28-05-2007, 20:57
I know that this is digging up an old issue (and probably a lot of angst :yikes: ), but Autodesk FIRSTbase recently sent out an e-mail survey asking for your imput on this season's awards voting and other services offered by FIRSTbase. I responded to the survery with a lot of the same viewpoints that I've espoused in this thread, and I'm just curious to see how many people have done the same and whether you think if there is a big enough response to do away with the voting or change some aspect of the competition whether something will change?
Mike Schroeder
28-05-2007, 22:04
My opinion is that Autodesk/FIRST should see if they can aquire volunteers from companies like Pixar or Dreamworks to judge the award on the championship level, i also think that they should use a pre-decided set of metrics released to teams at kickoff and that the award should be judged on technical stuff not which one has the prettiest colors...
=Martin=Taylor=
29-05-2007, 00:03
My opinion is that Autodesk/FIRST should see if they can aquire volunteers from companies like Pixar or Dreamworks to judge the award on the championship level, i also think that they should use a pre-decided set of metrics released to teams at kickoff and that the award should be judged on technical stuff not which one has the prettiest colors...
We are sponsored by DreamWorks and several of their animators mentor our team.
I'd be pretty happy if they were recruited to judge the AVA awards :D
Having a rubric would really help. But as with all movies, the real deciding factor will always lie with the storyboard and theme. If you go back and review the winning AVA's you'll see that they all flowed beautifully, and had great themes.
You can have technical excellence and a bad animation.
Robert Thacker-
29-05-2007, 14:07
I agree. That's something I've been trying to tell my team for years.
Evenstar_342
30-05-2007, 20:48
Professional judges would be a great idea! If Autodesk is concerned about the competition taking up too much of their time, that would be a great solution.
They had professionals in the past judge the awards, but getting professionals to judges from a company like Pixar may not be the best idea simply because the animation is not the most important part of the entry. You supposed to utilized Inventor into the design factor. What I've been thinking lately is to make a presentation mandatory for the top three teams, and you are simply permited computer files and a projector to display your product to a team of engineers how your system works. In the real engineering world this is how things are done.
Robert Thacker-
02-06-2007, 17:49
Now I like that idea!
I really like the idea of having FIRST participants vote, but obviously the team size issue is a problem. So here are my four ideas:
1. Limit voting to something like five per team, so that larger teams don't have a advantage, but also allows some individuality
2. Instead of voting "that one is the best," rank them from one to five.
3. DEFINATLY have the award at the regional level, as Hachiban VIII said "to give the little guy a chance"
4. I'm not sure if this is practical, but maybe move the award submision dedline back a few days. Many teams simply don't have enough members so that a few people can be fully dedicated to Inventor work during the build season.
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.