Chairman's Submission Inconsistencies

[Alex2614]

Originally Posted by BrendanB

Where are these posted?

Not surprisingly, it's somewhat hard to find if you don't know where to look.

http://www.usfirst.org/roboticsprogr...ard-winners%20

Originally Posted by Conor Ryan

I think a community sourced "Chairman's Audit Protocol" could be a good tool here. Maybe it could be referenced officially eventually. Setting a community wide standard for evidence of completed tasks would solve this issue provided judges or the community checked it at or before competition. I think the concept that teams do not disclose their submission, publicly, ahead of time also may make this concept worse.

I think FIRST is heading that direction, with the winners being required to record their presentation, and FIRST publically displaying essays, videos, and presentations for everyone to see. Granted, this is just the winners, but a step in that direction.
Also, I've noticed over the past couple years a change in the wording of the award criteria. IT has been hugely focused on "significant measurable impact" more and more. They want to see numbers and data more now than ever, and they will grill you for that information. I think FIRST is heading there, and I think there is more accountability now with the increased emphasis on numbers and also the increased emphasis on "the previous 3-5 years" instead of your team's whole history. I.e. we can't ride on stuff team members did many years ago.

[Conor Ryan]

Originally Posted by Alex2614

I think FIRST is heading that direction, with the winners being required to record their presentation, and FIRST publically displaying essays, videos, and presentations for everyone to see. Granted, this is just the winners, but a step in that direction.
Also, I've noticed over the past couple years a change in the wording of the award criteria. IT has been hugely focused on "significant measurable impact" more and more. They want to see numbers and data more now than ever, and they will grill you for that information. I think FIRST is heading there, and I think there is more accountability now with the increased emphasis on numbers and also the increased emphasis on "the previous 3-5 years" instead of your team's whole history. I.e. we can't ride on stuff team members did many years ago.

I agree! Its gotten better, but as you pointed out, this happens retroactively it doesn't help filter and select the winners. Generally if you have a quality judge advisor they help make sure the best teams win and they challenge the judges to do follow up research as necessary.

I also think a similar process could be put into place for other awards like EI too.

True independent evidence of the activities completed would be a big plus, referencing local newspapers, or hard evidence such as thank you letters from organizations teams help out would be an asset to an audit. You can always stage a chairman's video or pictures.

[Hot_Copper_Frog]

Over the years, I’ve run into what I perceive as inconsistent or falsified claims by teams. This ranges from content of Chairman’s essays, to what students are verbally saying to judges, to materials available in the pits or team websites. I really don’t think that it’s 100% intentional 100% of the time, especially when there is a lot of student or mentor turnover on a team in a short time period.

Just as it isn’t my job to police other team’s bag-and-tag procedures, it’s not my job to police their Chairman’s submissions. If I encounter something that’s blatantly false, then I’ll probably ask some questions to find out more. A lot of the time, I’ve misunderstood what they were trying to say, or it needed to be phrased differently. It’s irritating when I have a gut feeling that a team is “padding” their submission materials, but it’s really not my place to say anything.

In fact, it’s the judge’s job to dig sufficiently deep into a team’s materials that they feel comfortable with the information. For this reason, I have my students sit down, and go line by line in the essay and any other submission materials, and “cite” where the information is coming from. We build an internal bibliography of sorts, so that if we’re ever questioned on a claim or statistic, we can just pull the source from our evidence book. Having all of your numbers/achievements thoroughly documented saves a ton of time – it’s easily retrievable not only for the judges, but anybody else that may be questioning the integrity of our team’s work.

If teams are getting away with exaggerated claims or flat out lies, then it’s a problem with the system, not necessarily a problem with individual teams. The Chairman’s Award is supposed to represent the pinnacle of what it means to be a FRC team – you are a shining example to the rest of FIRST, and should be above reproach when it comes to your facts. I would certainly be in favor of a more rigorous fact-checking process, because I think it’s important to prioritize accountability. I think having an anonymous “tip” submission could go a long way towards helping this problem, as many teams don’t want to make themselves look bad by pointing fingers at somebody else and stay silent as a result. If the concerns could be handled at an organizational level and addressed through the judges, I wouldn't mind it. I would be concerned, however, about individuals taking advantage of an anonymous system with malicious intent to “ruin” or “sabotage” somebody’s chances. It’s certainly something that FIRST should discuss as a community, and maybe we can come up with some satisfactory solutions that would put most people’s worries to rest.

[cadandcookies]

Thank you so much for all the responses so far!

I've always been an advocate of assuming best intentions, especially in situations like this where all the facts aren't known by one (or sometimes both) parties. Outside of this year, I've been on both sides of this situation-- both the one someone asked about something they thought was wrong in our essay and someone who reached out to another team with questions about their essay. I know that I appreciated being asked directly about our submission, and the opportunity to both clarify the events in question with the person who asked and to make our submission clearer in intent, which is why I personally would lean towards directly asking the team about whatever is in question.

[EricH]

My proposal would be that you ask the team how they accomplished X task that seems improbable. And if there's a reasonable explanation or crossed wires somewhere, that allows them to explain or uncross. I would say that 99 times out of 100, that's the whole problem.

But if there's something really, really, impossible to explain/ignore (I've heard rumors of teams barely doing something and winning, while the team that set that thing up didn't win and was told they "copied" the other team!), then I suspect that FIRST needs to have one minor note added to the Manual. You'll notice that the Judge Advisor can be called on to answer process questions. My proposal on that "minor note" would be this: A team noticing major inconsistencies (I don't mean spelling/grammar, I mean very large exaggeration or situations like the above rumor, particularly if the team is unable to get a reasonable explanation from the team in question) in the "culture change" awards (RCA, EI) may leave a message for the JA at Pit Admin with a description of the inconsistencies, the team with the inconsistencies, and the team reporting. The JA would then presumably have the judges in question look into the report, and take appropriate action--which may go against the reporting team.

[Jacob Bendicksen]

I've run into this a little bit, and I completely agree with what's been said so far - ask the team in a nonjudgmental way to tell you more about it, and it'll become pretty obvious if it was stretched or not.

Also, as someone who has put more time that he should into multiple Chairman's essays, I'll say that this is a tricky thing when you're writing the essay. The line between 'casting something in the best possible light' and 'stretching it such that it isn't true' can be a very fuzzy line, and it's not always immediately apparent when you're working on the essay. As a result, I'd encourage everyone to assume the best: I'd bet that 9 times out of 10, it wasn't intentional deception - just trying to make something sound good.

[BigJ]

Originally Posted by Jacob Bendicksen

I've run into this a little bit, and I completely agree with what's been said so far - ask the team in a nonjudgmental way to tell you more about it, and it'll become pretty obvious if it was stretched or not.

Also, as someone who has put more time that he should into multiple Chairman's essays, I'll say that this is a tricky thing when you're writing the essay. The line between 'casting something in the best possible light' and 'stretching it such that it isn't true' can be a very fuzzy line, and it's not always immediately apparent when you're working on the essay. As a result, I'd encourage everyone to assume the best: I'd bet that 9 times out of 10, it wasn't intentional deception - just trying to make something sound good.

This. Much to presenter's chagrin, we are usually continuously tweaking lines in our scripts for weeks after the draft is "finished" based on how it comes across in practice!

[philso]

Perhaps if the judges get some training in "job interview techniques" they can more easily detect when the truth has been stretched beyond the line.

I have also seen teams who undersell themselves. An FLL team we have been prepare for the World Festival was practicing their Research Project and Core Values and did not mention many of their amazing accomplishments such as getting over 1000 students to participate in a survey and providing instructional materials to teachers all over the world AND receiving videos of the teachers actually using the materials they provided. It seems that they were so focused on implementing the mission of FIRST that they forgot that the judging sessions were their opportunity to justifiably brag about their accomplishments.

[OZ_341]

What we always told our Chairman's team was that you had to treat the Chairman's submission process as if you were going to court.

For every claim you make, you must have EVIDENCE. Photos, video, quotations, clippings, or independent statistics. If we did something amazing but did not have evidence to back it up, we did not put it in our Chairman's Award. If we were making a good faith estimate, we would always go with the lower estimate and then cut it back another 10% for safety sake until it was clearly reasonable to all. It was better to go with a low number, than to be perceived as padding your submission. If a cool event was on the schedule but had not happened yet, we did not include it until it actually happened.

It was a high standard that was sometimes frustrating for the kids. Sometimes they did something really cool but had no record of it or they had an event planned that had not happened yet. But we would not include any claims unless there was clear evidence to back it up.

Teams need to find a standard that works for them, but I think that evidence is the key to credibility.
If I were a judge, I would politely ask teams for this evidence.

[Anupam Goli]

Originally Posted by philso

It seems that they were so focused on implementing the mission of FIRST that they forgot that the judging sessions were their opportunity to justifiably brag about their accomplishments.

That's pretty inspiring in and of itself.

[Al Skierkiewicz]

I agree with Oz above. I believe that teams that are attempting Chairman's have a different way of assessing the actions of their team. They make decisions based on what Chairman's team are expected to do. (of course we we will help, that is what a Chairman's team does. No, we would not do that, a Chairman's team doesn't do those things.)
I cannot tell you how often someone has come to me and told me they know a Team X is doing something or lying about this or that. When I investigate, 100% of the time, the individual has been wrong. They either heard it second hand, out of context or the misunderstood what was being said. Often individuals will hear something one way that was not close to the intent of the person delivering the information. For instance, a student walking by a pit hears "I use a 50 amp charger" and assumes the team is using that for robot batteries. I go and investigate to find that mentor was talking about his boat.
A Chairman's team is truthful, sometimes to a fault. We once turned down a Qaulity Award because the judge's description was for another team. So my default standard for a Chairman's team is to believe what they are saying. If it sounds far fetched, I would ask them to explain. If starting 20 teams with only 10 students sounds like a lot, maybe they are totaling the number of teams they have started since their first season, 15 years ago. That sounds a lot more doable doesn't it. I guess what I am saying is don't listen with a closed mind. What you hear is not always what was said.

[angelah]

Originally Posted by Al Skierkiewicz

If starting 20 teams with only 10 students sounds like a lot, maybe they are totaling the number of teams they have started since their first season, 15 years ago. That sounds a lot more doable doesn't it.

It could also just be what they focus on. We've started 24 FIRST teams in two years with a team of less than a dozen kids (and only about 6 of them have helped with it.) We have had direct training and support for those teams, too, not just "Hey, you should start a team, good luck with that." We know it sounds far fetched, so our Chairman's video includes superintendents and principals speaking on our behalf. Our team has two main purposes - spreading FIRST teams and training/inspiring the students on the team - so it's just that we put a lot of time into those areas.

When we won Chairman's at a district this year, the judges came to our pit on Saturday morning for clarification, and we were able to show them pictures to prove our facts. I didn't mind at all, and the students thought it was great they had done something "unbelievable." We never expected a Chairman's win, because we don't do things to win it; we have the students present for it because it is good for them.

[gblake]

Wow,

Four things struck me as I read the posts here, the foruth is probably the most important.

1) The notion that I need a community to tell me what to do in situations like this is completely foreign to me. Like the cliche says, "There are two kinds of people in the world, those that ... and those that don't."

2) People saying that it's not Graciously Professional to expose a fraud, if they are aware one has been perpetrated. My reply, "Poppycock!"

3) People saying that it's not their place to expose a fraud, if they are aware one has been committed. My reply, "Of course you have a duty to expose frauds. Do not turn a blind eye when something harmful occurs in your community."

4)

My proposal would be that you ask the team how they accomplished X task that seems improbable. And if there's a reasonable explanation or crossed wires somewhere, that allows them to explain or uncross. I would say that 99 times out of 100, that's the whole problem.

The fourth and most important point is to be biased toward believing, to investigate with a positive attitude, to talk to people with different roles and responsibilities to check facts, and then if you are fully certain that a problem exists, bring it up discretely (especially if you are young, and perhaps have a hard time imagining just how little of the world you have experienced so far).

Blake

[OZ_341]

Originally Posted by gblake

Wow,

Four things struck me as I read the posts here, the foruth is probably the most important.

1) The notion that I need a community to tell me what to do in situations like this is completely foreign to me. Like the cliche says, "There are two kinds of people in the world, those that ... and those that don't."

2) People saying that it's not Graciously Professional to expose a fraud, if they are aware one has been perpetrated. My reply, "Poppycock!"

3) People saying that it's not their place to expose a fraud, if they are aware one has been committed. My reply, "Of course you have a duty to expose frauds. Do not turn a blind eye when something harmful occurs in your community."

4) The fourth and most important point is to be biased toward believing, to investigate with a positive attitude, to talk to people with different roles and responsibilities to check facts, and then if you are fully certain that a problem exists, bring it up discretely (especially if you are young, and perhaps have a hard time imagining just how little of the world you have experienced so far).

Blake

Good Thoughts.