[FRC Blog] Einstein Report Released

[Billfred]

Originally Posted by Greg McKaskle

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle

This. This. A million times, this.

If you're clever enough to figure out things like this, you should be smart enough to realize you have a duty to keep this knowledge out of the wrong hands and disclose it properly.

[JamesTerm]

Originally Posted by techhelpbb

The trick with the phone wouldn't have worked on at least 2 of the robots because those 2 had the B version of the D-Link AP on them at the time.

Ah HA! Those two robots must have been on the Blue alliance!

[techhelpbb]

Originally Posted by Greg McKaskle

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle

In the report it says this:

Page 10:

"While the Einstein matches were in progress, an individual was observed near the field using a cell phone in an apparent attempt to access the field WiFi network. This individual had attempted to engage field personnel in discussions while the field personnel were troubleshooting other issues. This individual was asked to put away the cell phone, and complied. Later, the individual was observed using the cell phone again, and at that point, before the last two Einstein matches were played, was asked to leave the field area, and did so."

"After Championship, this individual came forward wishing to share knowledge regarding the failed client authentication issue. The individual claimed to have attempted to connect to the network associated with Team 2056 during Semi-Final 2-1 and observed that this attempt corresponded with the robot losing communication."

Posts were made with slightly more information. That was pages ago and I'll leave that to someone else to cull.

I would have to suspect that the attempt to engage field personnel was their initial attempt to report the issue.

Originally Posted by Billfred

This. This. A million times, this.

If you're clever enough to figure out things like this, you should be smart enough to realize you have a duty to keep this knowledge out of the wrong hands and disclose it properly.

Please be aware that there are currently several vectors I could shut down: all the robots, some of the robots, or halt any of your robots at will.

Remember I do computer security and it's part of my job and no one asks me my opinion about this at FIRST.

Additionally I know other people have reported vectors some of which are not listed and have yet to get a clear line to FIRST either.

Your assumption presumes that you can get the ear of the person you need to talk to and that in reporting it someone decides to mitigate it.

So for example perhaps someone tried to be the 'white hat' when they had a person face to face at Einstein and didn't like what they got for it (I presume only...it could also be that they were just trying to cozy up to field personnel to get better signal strength).

Originally Posted by JamesTerm

Ah HA! Those two robot must have been on the Blue alliance!

Appendix A of the report is a little tricky to read because of how it's arranged in the left column.

Teams 16, 25, and 207 had the B version and would not have been effected by the stunt with the phone trying to connect.

[Greg McKaskle]

I'm sorry for paraphrasing, but at least one of the field personnel was an NI employee acting as CSA. No, it wasn't me, but I've heard this story from several points of view. I have to believe that there are better ways to gain signal strength. If you have access to Bill Miller, Frank Merrick, the FTAs, and virtually all of the FIRST staff wearing red ball caps, I think you can increase signal strength in many ways.

Greg McKaskle

[Denise Bohnsack]

It appears Marty McFly's Time Machine/Delorean would be useful now. It would be truly wonderful to go back and change these events and fix them before they had a chance to happen. Instead we are left with trying to prevent them in the future. I will leave this to the programmers and engineers.
I have followed this thread all week. For good reason, there is much anger and those effected have every right to feel this way for many reasons. As more than 1 poster has stated, this was a chance to have a blue flag for once. Just an idea to consider, may be a really bad one, but I will put it out there anyway just from a Mom's perspective. Could a flag/banner be created for this year's finalists that is extra special/limited edition/special color variety? Given to all that were in the finals since each might have had a chance at the title. Sort of a badge/banner of honor, for surviving and enduring the 2012 catastrophe? As the years go by, it would be special as no other team will have the chance to have such a banner, and the teams that go on can take pride if the fact they persevered and moved forward. I am in no way suggesting this makes up for what happened or replaces a title championship. I am suggesting this pays tribute to the teams who have endured much and are to be respected.
It is a small gesture but sometimes small gestures are very meaningful. I understand there would be an extra cost issue here which would need to be addressed to provide these teams with these banners, but I bet many of us who feel for these teams would gladly help. Again, just a Mom's suggestion and if some of you find it insulting or inadequate please forgive me, I promise I mean well. The only way to ever make a positive out of a very bad situation is to learn from it or make it better for those coming along next. Wish I had the perfect words or solution, but nobody does. FIRST will continue and let's try to make it better for future FIRSTers.

[techhelpbb]

Originally Posted by Greg McKaskle

I'm sorry for paraphrasing, but at least one of the field personnel was an NI employee acting as CSA. No, it wasn't me, but I've heard this story from several points of view. I have to believe that there are better ways to gain signal strength. If you have access to Bill Miller, Frank Merrick, the FTAs, and virtually all of the FIRST staff wearing red ball caps, I think you can increase signal strength in many ways.

Greg McKaskle

If in this case you equate 'signal strength' with finding someone who will listen then the question is how in someone's face can you get during a crisis (there were other issues at work). I don't know the particulars of the venue that day. I was at work in NYC watching over the Internet.

In this case I meant literal signal strength. Their ability to connect to the network over the any possible interference from other people would be improved by proximity to the field. Something you might be able to get by giving yourself a way to get close to it (say by talking to field personnel).

No matter what if they attempted to report it at all it would mean they tried it before that or saw someone else do it.

More importantly some of these attacks mirror the field troubles we noticed at Monty Madness off season the year before.
That was another example where I personally checked the D-Link AP and the robot and there were no problems.
Go out on the field and poof...

I did mention that in the topic about alternate control systems where you explained about the new logging features before the start of all the 2012 events.

I suspect this has been brewing for a while.

[Andy Baker]

Kudos to FIRST for the attention to detail, thoughtfulness, and completion of this report. The effort of FIRST, the volunteers and suppliers, and the Einstein 12 is much appreciated.

I really can't understand quite how the teams on Einstein feel about all of this, but I can understand the frustration, bewilderment, and anger. While the investigation is going to make things better, this situation is still tough to handle and is very frustrating. The sabotage was not just to certain teams, but to the entire FRC program and community.

Everyone builds their robots uniquely, runs their teams differently, and prefers all kinds of flavors of ice cream. Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Andy B.

[Barry Bonzack]

Originally Posted by Andy Baker

Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Disclaimer before clicking the below link: The youtube video contains some foul language. I copied at a time stamp past that portion of the TED talk.

Been following this thread, with a TED talk by Anthony Robbins ringing in my head. Everyone reacts to all situations differently. Some people will find silver linings to anything. Some people will point fingers. Some want heads to roll, others want to give a hug.

We are role models. Not just our mentors, our entire community to the rest of the world. However you react, make sure whoever is watching you is inspired by it in a positive way.

[Brandon_L]

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

[Steven Donow]

Originally Posted by Brandon_L

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

I'm obviously not affiliated with them, but I believe at regionals(CT and Alamo specifically) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

[MikeE]

Originally Posted by stevend1994

I'm obviously not affiliated with them, but I believe at regionals(CTand the other one that they went to that wasn't Lone Star... Alamo?) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

The CT eliminations (in week 5) saw multiple robot loss of control events from several good teams, some of which were clearly robot system issues (IIRC faulty battery connector, USB hub disconnection) but also others which had no clear cause identified at the time.

As Steven suggested it's possible that 118 suffered from the same subtle and unfortunate system interaction bug that was discovered before the Einstein investigation weekend.

But one of the effects of the Einstein revelations on us all as a community is that now there has to be some suspicion that a similar act of interference occurred at Hartford.
Innocence lost...

[Greg McKaskle]

I did not attend CT, but worked with 118 the following week in Houston, trying to identify the issue. I looked at the logs from CT and they looked very similar to Einstein. It is very likely that the sensor connection led to the failure in Houston and the ones in CT as well. They made code changes in Houston, I don't believe they did so before Einstein, and I don't believe they introduced the problem between divisions and Einstein. The code issue was present, lurking for a long time.

If the sensor connection had never failed, the loop in the init code would do what it normally did and the robot would have operated wonderfully.

If the sensor connection had failed permanently, they would have hooked up a complete debugger in the pits, located the loop and the sensor, fixed them both lickity-split, and operated wonderfully afterwards.

But the sensor connection apparently failed just a few times during the season. Perhaps it was brought on by the cart or the loading or reset procedure, or vibration, but since it didn't stay in a failure state, the chance to debug was fleeting. Additionally, the sensor wasn't used and the init code wasn't executed unless the testing included the auto-tele transition. I believe this was another factor that influenced how team 118 interpreted the cause. Bugs that are difficult to reproduce are incredibly frustrating in all disciplines.

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well. Most things in the world don't work the first time, they interact in ways you didn't predict, and they may change over time or under different conditions. Managing that chaos is a part of what engineers do.

118 is a great team that builds great robots, but this time both Murphy and Achilles had their influence. I look forward to working with them in the future.

Greg McKaskle

[JamesTerm]

Originally Posted by Greg McKaskle

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well.

Yes I agree... we have a completely different simulation to run our code. To add to this... I never... never write an infinite loop (and hope for some breaking condition). There should never be a case for that, but I see programmers do it (even some of the best).

Just to be clear... I define an infinite loop as this:
while (true)
{
if (I hope this works) break;
};

You can always do this instead

while (timeout++ < threshold)
{
if (I hope this works) break;
}
if (timeout >= threshold)
{
assert(false);
error recovery here;
}

Also any thing inside the autonomous loop (in this case) could check for
(IsAutonomous() && !IsDisabled())
to determine if autonomous is still happening... in my previous entry it is absolutely critical to exit autonomous loop ASAP... otherwise you'll lose telop connection for a good chunk of the match. (This is worth repeating).

If anyone knows of that issue being fixed please let me know... According to this report it sounds like it still is the same as it was last year.

[Arefin Bari]

After reading through the Einstein report and this thread several times; I had to digest all the information before making a post.

It is very unfortunate what happened on Einstein. I was home watching as my wife was screaming at me because we were supposed to get to dinner. I was very excited because most of the team that were on Einstein this year are my “friends” and they have done enough for team 108 for the last several years. I was ready to watch them tear it up on the field instead I sat there staring at dead robots.

I have nothing more to say than I am disgusted by the individual’s action. The teams affected have every right to be upset and frustrated. Heck, I still hold grudges against a mentor that came into our pit couple years ago and thought it would be gracious to disconnect all the pwm cables from the jaguars. It is unfortunate that situations like this happen in our community. I bet the Einstein teams attending IRI this year will bring their A game and I can’t wait to see them all compete in person.

[Justin Ridley]

Originally Posted by Brandon_L

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

We did not change any code before Einstein. As Greg states, this was likely an issue that was with us all season. While we know the gyro reporting bad data was the start of our error chain, we’re not 100% on why we got that bad data. The way we could replicate this failure post Einstein was by unplugging the gyro (no data = bad data). The report states a faulty crimp is the most likely cause. This may in fact be true, however, it’s a little hard to accept due to seeing the failure so infrequently and having no issues in between. For instance, we saw one of these failures on practice day of Houston. Without touching the robot, we were able to run another practice match immediately after with the exact same robots on the field. This match ran without issues.

After this practice match in Houston, we replaced the cRIO. We didn’t see another occurrence of the problem until the first match on Einstein. Obviously we thought the cRIO fixed the problem and there was no need for further troubleshooting.

Looking back we can only assume that the problem outlined in the report was the cause of the one failure we had practice day in Houston and the two semi-final matches in Connecticut. The second leg of the error chain was part of the code. After Connecticut, our programmers literally spent hours looking for a spin loop without an exit. It simply wasn’t obvious and they didn’t find it.

As Greg alluded to, one big lesson learned for us is how to better perform our full robot checkouts, which we do before and after every match. We did NOT include a run through autonomous mode during these checkouts, which meant we would never have seen this failure. Had we done this, we may have seen the failure more often and been able to better diagnose it.

We learned some valuable lessons from this, and are very thankful to Greg and the FIRST team up in NH for their work with us and all the Einstein teams.

We know how horrible it feels to have our robot not compete due to reasons that ultimately were our fault. We’re sure it must feel much worse to those teams who were brought down through no fault of their own. We can’t begin to express how saddened we are to know that this happened to our fellow competitors.