[FRC Blog] Einstein Report Released

[Alan Anderson]

Originally Posted by techhelpbb

FIRST's deauth vector is not new, Hack-A-Day exposed this very publicly last year and other sites well before that. All that was required to breach this? Download code.

I don't think you understand what the actual problem was. The system is indeed vulnerable to a deauthentication flood, or even a fast trickle. However, there were no such attempts detected, and there is no evidence to suggest that any occurred. The testing did show that it was possible to disrupt the connection without triggering a warning, so the detection parameters need to be tweaked to something more appropriate to the FRC use case. Still, it doesn't look like this was something that actually happened during competition.

The confirmed problem was instead an unknown and unexpected bug in the access point firmware that broke the existing connection when another client tried to authenticate and failed. Nothing special needs to be downloaded in order to cause this bug to be expressed.

[JB987]

Originally Posted by Holtzman

We in no way blame the entire team for the actions of this individual, but do feel they should stand up and acknowledge that a member of their team was responsible for the FCA attacks on multiple Einstein teams, and potentially others at the Championship and other events. We will give them time to do properly while being respectful to the innocent members of their team.

+1

[apalrd]

While we're sharing dead robot videos, how about

-67 in MSC SF1-2 http://youtu.be/ZbTHqBdvgJc - Clearly dead for the remainder of the match, blinking RSL.

-67 in MSC F-2 - This to me shows symptoms that could also be described as a loose ethernet cable. Team 67 dies in the key, and recovers when they are hit by team 469 (~15secs dead based on video). They then die later in the match, near the opponents bridge, and do not recover. I will share it anyway. http://youtu.be/CDLzSMPyYsc
-We died as well in the SF's under similar circumstances. I will not share the video.

I (as a member of the drive team) was by the field during all of those matches. In the all cases, the FTA's blamed the issue on the radio, and team 67 replaced their radio with a spare after SF1-2. I talked with team 67 (and team 469, their partners) personally after/during this, and they very thoroughly looked over their machine and found nothing (to my knowledge).



As to all of my earlier rants on the control system, I have two points that I was trying to make:
-The system is too complicated or difficult to setup if so many of the highest level teams cannot get it right.
-The system, while vulnerable to an unknown bug, was also vulnerable to a known, relatively simple bug (deauth) and there was nothing in place to detect or log this properly.

[Ryan Dognaux]

Originally Posted by Holtzman

I attended the Einstein Weekend with Eric Mech, one of our graduating seniors. Eric never left the school before 9pm this season. He was our lead programmer and poured his heart and soul into the team. All the teams on Einstein were brought together and told about the FCA attacks at a meeting on Sunday morning of the Einstein weekend. We were told that FIRST had evidence that our robot was specifically targeted and disabled. I raised my hand and asked what evidence they had, and then Frank Merrick told me that the person had openly admitted to it. I got up and had to leave the room. I was furious. Eric stayed, and handled the news a lot better than I did. He graduates this year not knowing of what we could have achieved.

Thank you for posting this. This personal account is just an example of how one person and one team was affected by this, and it's heartbreaking. The person responsible for this owes, at the very least, a personal apology to every single team that made it to Einstein this year. You not only made FIRST look bad, you took away something that in all likely-hood these students, mentors, teachers, parents & sponsors will never get back - a fair shot at winning the 2012 Championship.

I really wish the reason behind all the issues were purely related to technological failures, because we can fix those. This is a people issue. People are much harder to fix. I'm encouraged after reading the report that the technology-aspect of this competition will improve for next year's season. Let's strive to improve as people and as a community as well. No one should have to go through what these teams went through ever again.

[Alan Anderson]

Originally Posted by Pat Hart

One question to ask when demanding the person to come forward and apologize is "Could I forgive this person?"

What was done is unforgivable.

What the person did is terrible and there should be just punishment and consequences, but this person does not need to be forever hated or despised.

I do not believe everlasting hate is justified, but I see nothing wrong with the person being forever shunned.

[Ekcrbe]

Originally Posted by 1075guy

Video here

RSL is clearly flashing while it's dead, suggesting the cRIO is still running. Symptoms match what the report says happens with FCA.

Also, 1717 in this video. Symptoms match again.

I understand how strange it is. I only threw that out there because it matched symptoms we saw in the shop once (with the practice bot, granted) while running it with our '11 bot.

Our lead programming mentor reviewed our logs, and they showed that we never lost communications with anything, indicating we "should" have been working the whole time. Strange indeed.

[EricH]

For everyone who is saying that the team should come forwards:

Do you REALLY want that team's members to be known as "the team that ruined Einstein 2012" for years down the road? In their careers, even?

Whether or not that is your intent, that is what will happen. The team will gain that reputation, like it or not, if they come forwards. No amount of "This was an individual, not the team, we as a team don't go that route" and "This individual is no longer part of our team" will stop it. If it was one of your students or mentors, and you put forth your team as their former team, could you handle that stigma for years to come, in addition to any other mis-informed and unfair ones you may already carry or aquire in the future? Or would you or your sponsors simply cut the team?

If the individual were to announce himself or herself by issuing an apology, while leaving team identity out of it, that's another matter. In that case, one person gets the stigma--again, they have to deal with it. But again, certain curious ones may probe deeper and discover the team. See above paragraph.

If this act was truly the act of a lone person, operating without the consent or knowlege of the team, then the best way to keep the team from being tainted is to keep the person's identity--and that of their team--a secret known only to FRC staff, and as few of those as possible. If, on the other hand, the team was involved in some way, then there are other measures that FIRST can impose, which they have not.


With that said: The report, as a whole, shows a thorough investigation of the Einstein problems. It was never meant to investigate all the comm problems in FRC this year--such a study would take a full year. However, it does provide a path forwards to eliminating comm problems, and better diagnostics in the future. I like the report, both as a very well-written piece of technical writing and as an explanation of what happened on Einstein, and also as a guide to places to start looking for other issues that may be plaguing teams.

[ratdude747]

A tad late to the party here, but here's my $0.02:

First, I agree with FIRST in how they handled things. Well done.

Second, shame on whoever hacked the field. That is anything BUT what I was taught was proper FIRST behavior. What they did is unacceptable. As for their affiliated team, I think that whether they reveal themselves or not is their exclusive decision. If the team was truly not directly involved the incident, then I could honestly care less about who the team happens to be.

Third, After reading the individual team sections of the report, I can now say that this proves the not even the best teams are perfect and ANYBODY can can make a mistake. Lesson: every team should double check their work.

No further comment.

[Steven Donow]

Originally Posted by EricH

For everyone who is saying that the team should come forwards:

Do you REALLY want that team's members to be known as "the team that ruined Einstein 2012" for years down the road? In their careers, even?

shortened for shortening reasons...

Ugggh. I can't agree any more with this post. Regardless of how much "gracious professionalism" people have, regardless of how kindhearted people in FIRST are, THIS IS WHAT WILL HAPPEN. Call me cynical, but this is just how people are.

The only circumstances under which a public apology should be posted are, in my opinion:

1) Completely anonymous, sent by individual to FIRST, then posted by FIRST.

2) Sent to the main contacts of the "attacked" teams, then posted by someone from one of those teams, anonymously.

3) Basically, anything that keeps the identity of the "attacker" completely anonymous.


Again, I would love to believe that the general FIRST community would be accepting of it, but no matter what, the team would be referred to as "the team that attacked Einstein"

[techhelpbb]

Originally Posted by Alan Anderson

I don't think you understand what the actual problem was. The system is indeed vulnerable to a deauthentication flood, or even a fast trickle. However, there were no such attempts detected, and there is no evidence to suggest that any occurred. The testing did show that it was possible to disrupt the connection without triggering a warning, so the detection parameters need to be tweaked to something more appropriate to the FRC use case. Still, it doesn't look like this was something that actually happened during competition.

The confirmed problem was instead an unknown and unexpected bug in the access point firmware that broke the existing connection when another client tried to authenticate and failed. Nothing special needs to be downloaded in order to cause this bug to be expressed.

Unless you have something more than what is in the presented information I think you assume that AirTight which we now know can not see this issue under the right circumstances is sufficient reason to assume it did not happen. I draw nothing from this report that indicates to me they have actual raw data to confirm that a deauth attack did not happen on Einstein or elsewhere and it clearly is a well known vector with tools that often allow setup to exploit that hole in AirTight. Also you can make this work even if AirTight can detect it when fixed.

The second problem. The one you have listed as confirmed may be far more practical to point at and say well they did it and it requires no special tools we confirmed it. I actually mentioned that aggressive attempts to gain WiFi access could lead to this pages back. So I have acknowledged it but I think that concern about this and finger pointing is sort of crazy. There's actually premade devices you can buy that will locate and crack WPA passwords. Anyone could have brought one and had it in their pocket. Anyone at any point in the competition could have tripped over this. In point of fact well before Einstein suggestions about the versions of AP hardware surfaced. There where options to deal with this including spare parts in dumb luck you may get another version. The assumption here is that this person targeted teams with a mind to carry out rigging. How could they be sure the teams in question would be effected? They certainly can't walk over and offer to swap the AP.

On the other hand having now personally setup and torn down a field twice and looked at how AirTight is used. I am positive that a person could easily disable a robot or robots at will. They will still be able to do that when AirTight is patched and the AP versions are upgraded. I view this confirmation as a way to distract from the larger issue.

[Racer26]

@techhelpbb:

The majority of teams were running the affected firmwares for the robot radios. You wouldn't have to be sure. It would affect most teams, and its unlikely you would have ended up with a full alliance of teams that couldn't be targetted in this way.

[Jay O'Donnell]

First of all I'd like to say that FIRST has done a remarkable job in handling this problem. I can't think of another organization who could pull together this kind of report and would do as much work as they could to figure out the conclusions that they did. Bravo!

As for the interference, the individual in question got what they deserved and that's that. Whatever team they may have been a part of has no responsibility for them if they were acting by themselves. I personally believe that the team should stay anonymous because whether we like it or not, they will become "the team who ruined Einstein". Which of course they shouldn't, but unfortunately, that's just how it works.

As for the teams that were interfered with on Einstein, I hope all of you make a soon return to Einstein, because what happened is extremely unfair and selfish. For those of you that were effected by it, the FIRST community is here for you. All of these teams have shown the utmost professionalism, I don't know if there are many other teams thy could do so. While 180, 25, and 16 were a fantastic alliance and definitely have all the credentials of a championship alliance, I still feel sorry for certain teams who were unable to perform because of an outside attacker that they had no control over.

I have no doubt that us as a FIRST community will be able to get through this rough situation, and that the teams involved remain as successful as possible.

My $0.02

[BrendanB]

Originally Posted by EricH

....

Originally Posted by stevend1994

....

I can't agree with these posts more. At first I was stunned (still am), I think what happened is repulsive, I think an apology is owed, but it shouldn't be public. It should be anonymous to the 12 teams involved.

Some say that the team should step forward before it comes out later on. What if the team doesn't know a member on their team committed this act? Imagine if this was your team. Yes an apology is owed but if identities are made public that team and person will go down in FIRST history and not in a good way.

[Ether]

Originally Posted by 1075guy

The majority of teams were running the affected firmwares for the robot radios.

Just for the record: for the radio, it was a specific hardware revision that was vulnerable (Revision A), not the firmware.

[Racer26]

While I would love to agree with EricH et al, I don't.

The people that make up FIRST are some of the smartest people on the planet. We can read between lines, look for evidence, and investigate. Many people already know who is responsible, and many more already know what team they were associated with. Lots of people will do their own investigation into what went on.

I recognize that some people will forever associate the team with the interference, HOWEVER, I think that with time (and not much more time at that), the information will come out from another source, and that will make the lasting impression much worse than if it is admitted to and apologized for by the team.

The report was released approximately 22 hours ago. Lots of people have already come to same conclusions I have.