Chairman's Submission Inconsistencies

[cadandcookies]

Normally I'd post this in the Chairman's Award section, and I completely understand if it gets moved there, but I feel like there's a very important discussion that needs to be had now that winning Chairman's submissions are public, and that question is this:

How do you proceed if you read through a Chairman's essay and notice things that are inconsistent or obviously false?

This question came to me as I was finishing up reading through all the winning regional Chairman's submissions. I'm not saying that I noticed any particular team that had this issue, but to my knowledge, there are no FIRST-provided guidelines for dealing with this sticky situation, which means it's up to us as a community to figure it out until such a time as FIRST gives us guidelines.

How would you react? Who would you contact-- the team, or FIRST, or both??

[BigJ]

I think it all depends on the severity and (perceived) intent.

I know every year no matter how hard we proofread, there are always inconsistencies that get through to our pit boards (which means we re-print them for every competition).

I get really nervous about it -- In fact, one year that we were awarded a RCA the "judge snippet" listed at least one thing (not major, but part of the list-off) that we did not do and I am 100% sure (and confirmed) that none of our materials or students told or implied that thing to judges. I'm sure some wires got crossed in the judge room, but it was really concerning at the time.

I think in the end it's just like Stop Build Day, unbag windows, withholding allowance, pre-build restrictions, etc. We all operate under a giant honor system, and trust each other not to break it.

I know sometimes my students will write things that start to stretch the truth, but we bring it back. I find that the problems usually come from things 3+ years ago that current students have had little to no intaction with, which is why we are leading an effort this summer to create a "comprehensive" team encyclopedia so that we can be sure all our statistics are as accurately reported as possible.

[Alex2614]

Originally Posted by cadandcookies

Normally I'd post this in the Chairman's Award section, and I completely understand if it gets moved there, but I feel like there's a very important discussion that needs to be had now that winning Chairman's submissions are public, and that question is this:

How do you proceed if you read through a Chairman's essay and notice things that are inconsistent or obviously false?

This question came to me as I was finishing up reading through all the winning regional Chairman's submissions. I'm not saying that I noticed any particular team that had this issue, but to my knowledge, there are no FIRST-provided guidelines for dealing with this sticky situation, which means it's up to us as a community to figure it out until such a time as FIRST gives us guidelines.

How would you react? Who would you contact-- the team, or FIRST, or both??

I have witnessed this multiple times before, unfortunately. But I choose to leave things alone. Pardon me if this seems too political, but any time any of us calls someone else out on something like that, it only looks bad on us, especially if it cannot be proven. Not to mention that any attempt to do so will be considered "Un-GP" by many.

I do think, though, that we should have some kind of guideline in place. More extensive "background checks" on the teams might help, but you're cutting into more volunteer hours and it may just not be worth it. But like I said, I've seen things this multiple times, and ultimately the best thing to do on our end is just be adults about it and move on.

It's really unfortunate that this happens.

[PayneTrain]

Originally Posted by cadandcookies

Normally I'd post this in the Chairman's Award section, and I completely understand if it gets moved there, but I feel like there's a very important discussion that needs to be had now that winning Chairman's submissions are public, and that question is this:

How do you proceed if you read through a Chairman's essay and notice things that are inconsistent or obviously false?

This question came to me as I was finishing up reading through all the winning regional Chairman's submissions. I'm not saying that I noticed any particular team that had this issue, but to my knowledge, there are no FIRST-provided guidelines for dealing with this sticky situation, which means it's up to us as a community to figure it out until such a time as FIRST gives us guidelines.

How would you react? Who would you contact-- the team, or FIRST, or both??

This is definitely a discussion I saw bubbling up publicly when FIRST said they would be posting complete wining submissions online. It will be interesting to see how teams with less-than-genuine information in a winning submission are dealt with by the community near them and abroad. Regardless of feedback changes for the 2016 submissions I am hoping for, I do hope they continue to post winning submissions. It could be difficult for a mentor to explain to a team why a submission that looks weaker by a lot of measures won over theirs, but looking at essays and videos of some Hall of Fame probables with such ease is an invaluable resource.

[Jon Stratis]

You know, I consider myself fairly knowledgeable about the local teams here. I know and am friendly with most of the mentors, I follow teams on Twitter and Facebook, I see posts by them on CD... Yet I wouldn't feel comfortable saying that anything in any chairman's essay from those teams is false. I just don't know everything about those teams, and can't know unless I'm actually part of the team and privy to their internal discussions and calendars.

If you come across something that strikes you as improbable, it's better to find a way to ask about it in an interested manor, than an accusatory one. Something like "wow, how'd you start up 30 FTC teams? That sounds like a huge amount of work!", versus " there's no way you can meaningfully contribute to mentoring 30 FTC with the 10 students your team has!". What you might find out, by approaching the team in a better way, is that they're doing something really useful, creative, and ultimately something that you may be able to adopt in the future!. And you know, if they are BSing about it... Then simply by drawing some attention to it by asking about it may help bring them back in line.

[OAXACA]

I have definitely noticed some teams have stretched the truth a little bit or a whole lot. But I agree, it isn't very GP to point out those teams very specifically. It is a very unfortunate thing that happens. I think it is up to the judges to crosscheck any gray areas. I do realize it that such things that will take time from our fellow volunteers and judges. But it is also up to the teams to be honest and make that choice to be GP. But for now, I think it's best to just let things go and hope that teams are more honest in the future.

[Conor Ryan]

I think a community sourced "Chairman's Audit Protocol" could be a good tool here. Maybe it could be referenced officially eventually. Setting a community wide standard for evidence of completed tasks would solve this issue provided judges or the community checked it at or before competition. I think the concept that teams do not disclose their submission, publicly, ahead of time also may make this concept worse.

This is similar to the vague financial reporting standards defined in GAAP accounting or the standards organizations like Charity Navigator check. Unfortunately Charity Navigator only audits organizations with over $500,000 in public support. If we got a few hall of fame teams to buy into this concept I think it would go far.

[BrendanB]

Originally Posted by cadandcookies

This question came to me as I was finishing up reading through all the winning regional Chairman's submissions. I'm not saying that I noticed any particular team that had this issue, but to my knowledge, there are no FIRST-provided guidelines for dealing with this sticky situation, which means it's up to us as a community to figure it out until such a time as FIRST gives us guidelines.

Where are these posted?

[Alex2614]

Originally Posted by BrendanB

Where are these posted?

Not surprisingly, it's somewhat hard to find if you don't know where to look.

http://www.usfirst.org/roboticsprogr...ard-winners%20

Originally Posted by Conor Ryan

I think a community sourced "Chairman's Audit Protocol" could be a good tool here. Maybe it could be referenced officially eventually. Setting a community wide standard for evidence of completed tasks would solve this issue provided judges or the community checked it at or before competition. I think the concept that teams do not disclose their submission, publicly, ahead of time also may make this concept worse.

I think FIRST is heading that direction, with the winners being required to record their presentation, and FIRST publically displaying essays, videos, and presentations for everyone to see. Granted, this is just the winners, but a step in that direction.
Also, I've noticed over the past couple years a change in the wording of the award criteria. IT has been hugely focused on "significant measurable impact" more and more. They want to see numbers and data more now than ever, and they will grill you for that information. I think FIRST is heading there, and I think there is more accountability now with the increased emphasis on numbers and also the increased emphasis on "the previous 3-5 years" instead of your team's whole history. I.e. we can't ride on stuff team members did many years ago.

[Conor Ryan]

Originally Posted by Alex2614

I think FIRST is heading that direction, with the winners being required to record their presentation, and FIRST publically displaying essays, videos, and presentations for everyone to see. Granted, this is just the winners, but a step in that direction.
Also, I've noticed over the past couple years a change in the wording of the award criteria. IT has been hugely focused on "significant measurable impact" more and more. They want to see numbers and data more now than ever, and they will grill you for that information. I think FIRST is heading there, and I think there is more accountability now with the increased emphasis on numbers and also the increased emphasis on "the previous 3-5 years" instead of your team's whole history. I.e. we can't ride on stuff team members did many years ago.

I agree! Its gotten better, but as you pointed out, this happens retroactively it doesn't help filter and select the winners. Generally if you have a quality judge advisor they help make sure the best teams win and they challenge the judges to do follow up research as necessary.

I also think a similar process could be put into place for other awards like EI too.

True independent evidence of the activities completed would be a big plus, referencing local newspapers, or hard evidence such as thank you letters from organizations teams help out would be an asset to an audit. You can always stage a chairman's video or pictures.

[Hot_Copper_Frog]

Over the years, I’ve run into what I perceive as inconsistent or falsified claims by teams. This ranges from content of Chairman’s essays, to what students are verbally saying to judges, to materials available in the pits or team websites. I really don’t think that it’s 100% intentional 100% of the time, especially when there is a lot of student or mentor turnover on a team in a short time period.

Just as it isn’t my job to police other team’s bag-and-tag procedures, it’s not my job to police their Chairman’s submissions. If I encounter something that’s blatantly false, then I’ll probably ask some questions to find out more. A lot of the time, I’ve misunderstood what they were trying to say, or it needed to be phrased differently. It’s irritating when I have a gut feeling that a team is “padding” their submission materials, but it’s really not my place to say anything.

In fact, it’s the judge’s job to dig sufficiently deep into a team’s materials that they feel comfortable with the information. For this reason, I have my students sit down, and go line by line in the essay and any other submission materials, and “cite” where the information is coming from. We build an internal bibliography of sorts, so that if we’re ever questioned on a claim or statistic, we can just pull the source from our evidence book. Having all of your numbers/achievements thoroughly documented saves a ton of time – it’s easily retrievable not only for the judges, but anybody else that may be questioning the integrity of our team’s work.

If teams are getting away with exaggerated claims or flat out lies, then it’s a problem with the system, not necessarily a problem with individual teams. The Chairman’s Award is supposed to represent the pinnacle of what it means to be a FRC team – you are a shining example to the rest of FIRST, and should be above reproach when it comes to your facts. I would certainly be in favor of a more rigorous fact-checking process, because I think it’s important to prioritize accountability. I think having an anonymous “tip” submission could go a long way towards helping this problem, as many teams don’t want to make themselves look bad by pointing fingers at somebody else and stay silent as a result. If the concerns could be handled at an organizational level and addressed through the judges, I wouldn't mind it. I would be concerned, however, about individuals taking advantage of an anonymous system with malicious intent to “ruin” or “sabotage” somebody’s chances. It’s certainly something that FIRST should discuss as a community, and maybe we can come up with some satisfactory solutions that would put most people’s worries to rest.

[cadandcookies]

Thank you so much for all the responses so far!

I've always been an advocate of assuming best intentions, especially in situations like this where all the facts aren't known by one (or sometimes both) parties. Outside of this year, I've been on both sides of this situation-- both the one someone asked about something they thought was wrong in our essay and someone who reached out to another team with questions about their essay. I know that I appreciated being asked directly about our submission, and the opportunity to both clarify the events in question with the person who asked and to make our submission clearer in intent, which is why I personally would lean towards directly asking the team about whatever is in question.

[EricH]

My proposal would be that you ask the team how they accomplished X task that seems improbable. And if there's a reasonable explanation or crossed wires somewhere, that allows them to explain or uncross. I would say that 99 times out of 100, that's the whole problem.

But if there's something really, really, impossible to explain/ignore (I've heard rumors of teams barely doing something and winning, while the team that set that thing up didn't win and was told they "copied" the other team!), then I suspect that FIRST needs to have one minor note added to the Manual. You'll notice that the Judge Advisor can be called on to answer process questions. My proposal on that "minor note" would be this: A team noticing major inconsistencies (I don't mean spelling/grammar, I mean very large exaggeration or situations like the above rumor, particularly if the team is unable to get a reasonable explanation from the team in question) in the "culture change" awards (RCA, EI) may leave a message for the JA at Pit Admin with a description of the inconsistencies, the team with the inconsistencies, and the team reporting. The JA would then presumably have the judges in question look into the report, and take appropriate action--which may go against the reporting team.

[Jacob Bendicksen]

I've run into this a little bit, and I completely agree with what's been said so far - ask the team in a nonjudgmental way to tell you more about it, and it'll become pretty obvious if it was stretched or not.

Also, as someone who has put more time that he should into multiple Chairman's essays, I'll say that this is a tricky thing when you're writing the essay. The line between 'casting something in the best possible light' and 'stretching it such that it isn't true' can be a very fuzzy line, and it's not always immediately apparent when you're working on the essay. As a result, I'd encourage everyone to assume the best: I'd bet that 9 times out of 10, it wasn't intentional deception - just trying to make something sound good.

[BigJ]

Originally Posted by Jacob Bendicksen

I've run into this a little bit, and I completely agree with what's been said so far - ask the team in a nonjudgmental way to tell you more about it, and it'll become pretty obvious if it was stretched or not.

Also, as someone who has put more time that he should into multiple Chairman's essays, I'll say that this is a tricky thing when you're writing the essay. The line between 'casting something in the best possible light' and 'stretching it such that it isn't true' can be a very fuzzy line, and it's not always immediately apparent when you're working on the essay. As a result, I'd encourage everyone to assume the best: I'd bet that 9 times out of 10, it wasn't intentional deception - just trying to make something sound good.

This. Much to presenter's chagrin, we are usually continuously tweaking lines in our scripts for weeks after the draft is "finished" based on how it comes across in practice!