[FRC Blog] Einstein Report Released

[Andy Baker]

Kudos to FIRST for the attention to detail, thoughtfulness, and completion of this report. The effort of FIRST, the volunteers and suppliers, and the Einstein 12 is much appreciated.

I really can't understand quite how the teams on Einstein feel about all of this, but I can understand the frustration, bewilderment, and anger. While the investigation is going to make things better, this situation is still tough to handle and is very frustrating. The sabotage was not just to certain teams, but to the entire FRC program and community.

Everyone builds their robots uniquely, runs their teams differently, and prefers all kinds of flavors of ice cream. Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Andy B.

[Barry Bonzack]

Originally Posted by Andy Baker

Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Disclaimer before clicking the below link: The youtube video contains some foul language. I copied at a time stamp past that portion of the TED talk.

Been following this thread, with a TED talk by Anthony Robbins ringing in my head. Everyone reacts to all situations differently. Some people will find silver linings to anything. Some people will point fingers. Some want heads to roll, others want to give a hug.

We are role models. Not just our mentors, our entire community to the rest of the world. However you react, make sure whoever is watching you is inspired by it in a positive way.

[Brandon_L]

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

[Steven Donow]

Originally Posted by Brandon_L

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

I'm obviously not affiliated with them, but I believe at regionals(CT and Alamo specifically) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

[MikeE]

Originally Posted by stevend1994

I'm obviously not affiliated with them, but I believe at regionals(CTand the other one that they went to that wasn't Lone Star... Alamo?) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

The CT eliminations (in week 5) saw multiple robot loss of control events from several good teams, some of which were clearly robot system issues (IIRC faulty battery connector, USB hub disconnection) but also others which had no clear cause identified at the time.

As Steven suggested it's possible that 118 suffered from the same subtle and unfortunate system interaction bug that was discovered before the Einstein investigation weekend.

But one of the effects of the Einstein revelations on us all as a community is that now there has to be some suspicion that a similar act of interference occurred at Hartford.
Innocence lost...

[Greg McKaskle]

I did not attend CT, but worked with 118 the following week in Houston, trying to identify the issue. I looked at the logs from CT and they looked very similar to Einstein. It is very likely that the sensor connection led to the failure in Houston and the ones in CT as well. They made code changes in Houston, I don't believe they did so before Einstein, and I don't believe they introduced the problem between divisions and Einstein. The code issue was present, lurking for a long time.

If the sensor connection had never failed, the loop in the init code would do what it normally did and the robot would have operated wonderfully.

If the sensor connection had failed permanently, they would have hooked up a complete debugger in the pits, located the loop and the sensor, fixed them both lickity-split, and operated wonderfully afterwards.

But the sensor connection apparently failed just a few times during the season. Perhaps it was brought on by the cart or the loading or reset procedure, or vibration, but since it didn't stay in a failure state, the chance to debug was fleeting. Additionally, the sensor wasn't used and the init code wasn't executed unless the testing included the auto-tele transition. I believe this was another factor that influenced how team 118 interpreted the cause. Bugs that are difficult to reproduce are incredibly frustrating in all disciplines.

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well. Most things in the world don't work the first time, they interact in ways you didn't predict, and they may change over time or under different conditions. Managing that chaos is a part of what engineers do.

118 is a great team that builds great robots, but this time both Murphy and Achilles had their influence. I look forward to working with them in the future.

Greg McKaskle

[JamesTerm]

Originally Posted by Greg McKaskle

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well.

Yes I agree... we have a completely different simulation to run our code. To add to this... I never... never write an infinite loop (and hope for some breaking condition). There should never be a case for that, but I see programmers do it (even some of the best).

Just to be clear... I define an infinite loop as this:
while (true)
{
if (I hope this works) break;
};

You can always do this instead

while (timeout++ < threshold)
{
if (I hope this works) break;
}
if (timeout >= threshold)
{
assert(false);
error recovery here;
}

Also any thing inside the autonomous loop (in this case) could check for
(IsAutonomous() && !IsDisabled())
to determine if autonomous is still happening... in my previous entry it is absolutely critical to exit autonomous loop ASAP... otherwise you'll lose telop connection for a good chunk of the match. (This is worth repeating).

If anyone knows of that issue being fixed please let me know... According to this report it sounds like it still is the same as it was last year.

[Arefin Bari]

After reading through the Einstein report and this thread several times; I had to digest all the information before making a post.

It is very unfortunate what happened on Einstein. I was home watching as my wife was screaming at me because we were supposed to get to dinner. I was very excited because most of the team that were on Einstein this year are my “friends” and they have done enough for team 108 for the last several years. I was ready to watch them tear it up on the field instead I sat there staring at dead robots.

I have nothing more to say than I am disgusted by the individual’s action. The teams affected have every right to be upset and frustrated. Heck, I still hold grudges against a mentor that came into our pit couple years ago and thought it would be gracious to disconnect all the pwm cables from the jaguars. It is unfortunate that situations like this happen in our community. I bet the Einstein teams attending IRI this year will bring their A game and I can’t wait to see them all compete in person.

[Justin Ridley]

Originally Posted by Brandon_L

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

We did not change any code before Einstein. As Greg states, this was likely an issue that was with us all season. While we know the gyro reporting bad data was the start of our error chain, we’re not 100% on why we got that bad data. The way we could replicate this failure post Einstein was by unplugging the gyro (no data = bad data). The report states a faulty crimp is the most likely cause. This may in fact be true, however, it’s a little hard to accept due to seeing the failure so infrequently and having no issues in between. For instance, we saw one of these failures on practice day of Houston. Without touching the robot, we were able to run another practice match immediately after with the exact same robots on the field. This match ran without issues.

After this practice match in Houston, we replaced the cRIO. We didn’t see another occurrence of the problem until the first match on Einstein. Obviously we thought the cRIO fixed the problem and there was no need for further troubleshooting.

Looking back we can only assume that the problem outlined in the report was the cause of the one failure we had practice day in Houston and the two semi-final matches in Connecticut. The second leg of the error chain was part of the code. After Connecticut, our programmers literally spent hours looking for a spin loop without an exit. It simply wasn’t obvious and they didn’t find it.

As Greg alluded to, one big lesson learned for us is how to better perform our full robot checkouts, which we do before and after every match. We did NOT include a run through autonomous mode during these checkouts, which meant we would never have seen this failure. Had we done this, we may have seen the failure more often and been able to better diagnose it.

We learned some valuable lessons from this, and are very thankful to Greg and the FIRST team up in NH for their work with us and all the Einstein teams.

We know how horrible it feels to have our robot not compete due to reasons that ultimately were our fault. We’re sure it must feel much worse to those teams who were brought down through no fault of their own. We can’t begin to express how saddened we are to know that this happened to our fellow competitors.

[Al Skierkiewicz]

Justin,
It is my understanding that your code has a method for checking all sensors as part of the initiation sequence. When the gyro reported bad, the code stopped. We tested this in NH and received the same result when the gyro wire was pulled. During Einstein, we replaced your Crio, DSC, DSC cable and everything else I could think of without effect. We checked for power issues, bent cables, as much as we could in the amount of time we had. I have to tell everyone that 1114 and others were offering hardware and assistance during this time as you would expect. (remember Thunder Chickens from a few years ago?) The FTA and CSA came over to lend support as well.

[Al Skierkiewicz]

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

[techhelpbb]

Originally Posted by Al Skierkiewicz

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

I suppose one should ask what they should expect for a response if they do this?

[EricVanWyk]

Do we have any volunteers for illustrator?

The weird part? This is how I actually think about the control system. Engineering is so much easier when you give all the components personalities. Then you just watch the drama unfold and try to fix it.

[IKE]

Originally Posted by EricVanWyk

Do we have any volunteers for illustrator?

The weird part? This is how I actually think about the control system. Engineering is so much easier when you give all the components personalities. Then you just watch the drama unfold and try to fix it.

So, would you consider yourself a group counselor?

[dyanoshak]

Originally Posted by IKE

So, would you consider yourself a group counselor?

Actually, I hear that EVW was an LCPC (Licensed Clinical Professional Counselor) for about 20 minutes last year...