[FRC Blog] Einstein Report Released

[Chris is me]

Firstly, thanks to everyone in and out of FIRST who made this exhaustive testing and report possible. It is great to have such a thorough analysis of the forces at play on Einstein and the lengths they went to replicate on field conditions were extraordinary. I sincerely hope this leads to dramatically less communications faults at any event this year.

It's absolutely appalling that someone in FIRST would sabotage an alliance (and then some) by exploiting a security vulnerability. It's also appalling that it could be so simple to knock a robot out of commission in any FRC match since Week 4. Cisco's got some 'splainin' to do...

Did this person interfere intentionally? Yes.
Do I believe that this person had malicious intent at heart? No.
Do I believe that the person has been adequately punished? Yes.

I'm having trouble processing the idea that someone could deliberately attack all of the robots on an alliance by exploiting a security vulnerability without "malicious intent". Perhaps they were trying to raise attention of an issue FIRST should have known about (just about the only scenario I can think of that would even resemble "good intent"). Einstein is by no means whatsoever the proper time and place to demonstrate this problem.

And does it even matter what his / her intent was? Are the affected teams supposed to feel better about being cheated out of a fair chance at victory because "oh, he / she had good intentions"?

[techhelpbb]

Originally Posted by Alexa Stott

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match. If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to? Because of those facts, I think it's hard to say that the intent was not malicious.

As the report points out a not trivial amount of problems were actually on the robot. This would make it hard to see cause and effect while testing such an attack. You wouldn't know if the attack stopped the robot or the robot stopped for reasons you had nothing to do with. Just like the FTA and other FIRST folks wouldn't know if the robots were broken or someone was causing trouble.

Add to this the storm and the other distractions it's just a perfectly bad combination.

[PayneTrain]

I know I need to let this all marinate, but if the cause is so noble there is a far better way to achieve the results without taking it out on an alliance that has quickly become a punching bag in darker corners of the community (even though a member of it is the newest team in the FRC HoF...)

:/ this is rough stuff

[connor.worley]

Originally Posted by 1075guy

This reminds me of a story told to me by one of my university profs in networking.

Some friend of his found a flaw in the City of Toronto's (IIRC, big metropolitan centre anyway) traffic light control system. Tried to tell city council and they ignored him, blew him off. So he turned every traffic light red in the city for several minutes. When they came after him? He said "Be thankful I didn't turn them green."

I think taking down Einstein is the equivalent of "turning the lights green" here though.

[Brandon_L]

Originally Posted by Alexa Stott

It's in Jon Dudas' statement (third paragraph):

The poster he is referring to stated that non of the Einstein teams were involved, not none of the winning alliance.

[Alexa Stott]

Originally Posted by techhelpbb

As the report points out a not trivial amount of problems were actually on the robot. This would make it hard to see cause and effect while testing such an attack. You wouldn't know if the attack stopped the robot or the robot stopped for reasons you had nothing to do with. Just like the FTA and other FIRST folks wouldn't know if the robots were broken or someone was causing trouble.

Add to this the storm and the other distractions it's just a perfectly bad combination.

S/he was asked to stop and did not comply. The person also attempted to inform officials about it, indicating (to me, at least) that s/he was aware that what they were doing could have been causing issues.

If it had been one match that this happened in and had the person put the phone away when asked, my feelings on this would be much different.

[techhelpbb]

Originally Posted by Alexa Stott

[bS/he was asked to stop and did not comply[/b]. The person also attempted to inform officials about it, indicating (to me, at least) that s/he was aware that what they were doing could have been causing issues.

If it had been one match that this happened in and had the person put the phone away when asked, my feelings on this would be much different.

Then again putting the phone away doesn't mean much if you think about it. I can leave an application running with my phone screen off on any Android device.

So they put it in their pocket and it's still causing trouble. Then they take it out and it's still causing trouble. They are likely just as distracted as everyone else with the weather, the people watching, people's behavior near them, etc. It's really hard to say what people do when they put themselves in a bad spot like that. From the second they were asked to put it away they were in fact in a lot of trouble.

I don't disagree with your gut reaction. After all I had concerns about power issues before this all started and FIRST suggested we not test at Einstein those issues. As it turned out some of those issues existed who knows maybe they would have been found (I can't prove it either way). I even went so far as to ask the question in the official forum with regards to the championship.

However, no means no. We completed the testing of my little oscilloscopes for the power to the radio on off season events. We did it where it would do as little harm as possible to FIRST proper if something happened that was not expected. We did it with full knowledge of those who could be effected.

There was most definitely bad judgement at work with this person's choice of actions. They've made themselves a sitting duck for suspicion. It's highly unlikely we will ever know if they are the only source of the deauth issue.

The amateur way they handled themselves however, concerns me because usually where there's an amateur trouble maker that is easy to spot it is a distraction for someone that's not an amateur.

I mean not to start a witch hunt just to make the point. We live in a complicated world where rarely do specific effective security issues just magically appear to a single person. Like the systems themselves they are collaborative efforts. I have a hard time believing that a person so clearly asking to get caught just figured this all out on their own.

In a perverse way the drive to seek out the problem with the WiFi might have guided them to this issue via collaboration, but even then I doubt they crafted the attack and were ready to report it just in time of that one event.

I do computer security for a living in part so perhaps it's just in my nature to rarely assume the singular genius that hands themselves to me on the silver platter isn't covering for someone else.

[Ether]

Imagine the chaos if FIRST allowed any random person to walk in and give their suggestions while trying to diagnose the issue during Einstein. They'd be flooded with every person out there who has a theory (take a look at the numerous Einstein threads filled with people bickering about what they think happened to get an idea of how many people that would be) to step in.

Alexa, I think you are misinterpreting the previous posts. No one is suggesting that any random person should be allowed to walk in and give their suggestions while the FTAs are trying to diagnose a field issue.

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match.

The document says that there was a witness(es) whose description of a person using a phone matched the description of the person on the field earlier. Perhaps it was the same person.

If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to?

We don't know.

[Alexa Stott]

Originally Posted by Ether

The document says that there was a witness(es) whose description of a person using a phone matched the description of the person on the field earlier. Perhaps it was the same person.

FIRST officials spotted him twice:

This individual was asked to put away the cell phone, and complied. Later, the individual was observed using the cell phone again, and at that point, before the last two Einstein matches were played, was asked to leave the field area, and did so.

[Hjelstrom]

Originally Posted by Karthik

FIRST Hall of Fame Team 1114, Simbotics, would like to thank FIRST for thoroughly investigating, addressing and documenting the robot failures that took place on the Einstein Field at the 2012 FIRST World Championship. We would especially like to thank Frank Merrick and his staff for constantly communicating with us through the process.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

+1 to this. The honorable way for the team involved to handle this, even if it was a rogue member of a team acting alone, would be to come forward and spare this community from speculation and suspicion for years to come. You're protecting yourselves at the expense of many other teams. It won't work.

[Joe Johnson]

My head spins there is so much to take in.

First, I join my voice to the chorus that is saddened and outraged by the behavior of this individual*. I don't suppose that this is the first malicious attack by an individual on another FIRST team or alliance but this attack is so brazen. I am sick. Really sick.

Second, I don't think that it is too much to expect the former team of this individual to apologize to the teams involved as well as the entire FIRST community. What is more, depending on the circumstances, I would think that it may be appropriate for them to assure the FIRST community that they are committed to Gracious Professionalism in its highest sense and that they are committed to rooting out anything within their team culture that may have contributed to fostering the behavior of this individual.

I don't want to start a rumor but does anyone else think that this must be related to the nonsense that went on at the Greater Toronto East Regional? If so, the Canadian FIRST community really has to work to lance this boil.

Third, I am really disturbed by this statement in the report:

FMS White Paper – FRC will be producing a white paper which describes how the Field Management System operates. This will include details on the topology of the system, components used and the communications paths between the various components.

This seems to imply that they don't have this now. Wow.

Fourth, they offer "Additional emphasis in training and documentation" as a mitigation for "Robot D-Link radio reboot due to power dip." I hate this proposal. It seems to me that either the robots must return to active duty (passing packets back and forth) in a handful of milliseconds OR the radios must work down to a voltage that is almost literally impossible to get to without tripping the breaker (say 1.5 volts or something ridiculous like that).

Fifth, it seems to me that FIRST (and the FMS) has one implied contract with the teams: We will get X% of your data packets from your Operator Interface to your Robot and vice versa within Y msec. In my view of the world, literally the SECOND the FMS breaks this contract for even one robot on the field, the match has to stop and be replayed. Period.

With the current system this vision cannot be fully implemented because of dead/disconnected batteries and other complications.

I will say that if FIRST had this as their standard, this attack would not have been prevented but the attacker's purpose would not have been accomplished, which may have kept the attacker from even trying.

Reading the report, it is hard to know if the FMS system actively monitors this contract... ...past is past. Going forward, I really think FIRST should implement such a system hosting another tournament.

Finally, I hope that the FIRST community can pull together as a result of these events. We must do our best make FIRST better because of it. I have faith in our ability to do so.

Regards,
Joe J.

*I say "individual" because I am not sure if there were two or one. The report seems to indicate two folks were involved but only one was banned for life so... ...perhaps I am misunderstanding the text.

While this is a terrible event (it's terrible, it happened, end of story), my concern is that field issues happened all around the world during regionals/districts. Was a similar event the cause of all of those disruptions? There are people in the world who sometimes make the wrong decision, but to have such a person at all of the events where connection issues were present doesn't seem right.

[kazzykaty]

I think that FIRST did a great job at handling this situation. This is something that could have easily been ignored or swept under the rug. No one would have known. Instead, they put a lot of time and money into the investigation.

It is a shame that something like this would occur in a student-based organization but it is comforting to know that FIRST puts its participants and ethics first.

I am so proud to have been a part of such a great organization.

[Tom Line]

Originally Posted by techhelpbb

The larger issue than who did this is how was the system allowed to get to the point that it was possible at all. Let's worry about all the other D.O.S. (denial of service) attacks we have yet to find.

Clearly changes need to be made. It took extraordinary effort on the part of too many people to resolve even these issues to this point.

It still goes back to the assumption that the system is above flaw and that assumption being incorrect. In this case the system has a security issue and an active exploiter. Take if from me: you can look for and fix security issues before they get exploited as best you can or you can wait until they cost you reputation, resources and opportunity. Had they even profiled the issue before hand they could have dramatically reduced the chaos after the fact (if you don't fix it at least acknowledge it exists at the remediation level).

I disagree entirely. I don't believe anyone believes (or believed prior to Einstein) that the system is above flaw.

Take any system, no matter how well designed, and subject it to 60,000 ambitious folks all playing with it and see how secure it is.

This week's 'Yahoo' password hack displays just what happens when even the most competent network security is open for public interaction. Someone WILL find a way in. Google, Microsoft, and even the stock market have been subject to security invasions as well.

I hate to say it, but in this situation security through obscurity is FIRST's best bet. The entire system needs to be removed from the consumer electronics spectrum that all these common tools are designed to work with. I.e. - standard a/b/g/n wireless needs to disappear. If this does not change and go to a proprietary system, I will 100% guarantee you WILL see this happen again.

[akoscielski3]

Originally Posted by Karthik

FIRST Hall of Fame Team 1114, Simbotics, would like to thank FIRST for thoroughly investigating, addressing and documenting the robot failures that took place on the Einstein Field at the 2012 FIRST World Championship. We would especially like to thank Frank Merrick and his staff for constantly communicating with us through the process.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

I actually would prefer this team remain anonymous. Reason is that the Team itself is not responsible (or so it seems). And what ever the mentor/student/parent did wont ruin the whole teams reputation. If the team is revealed the team will become ashamed for what another individual did and could possibly shut down. I would hate to see this happen. Its not the Team's fault and I would hate to see them suffer from it.

Just my $0.02 CAN