[FRC Blog] Einstein Report Released

[Chinmay]

Originally Posted by Al Skierkiewicz

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

I think what Nick might of meant, and what I personally am thinking about isn't when FIRST discovered the bug, but when the individual discovered that he/she was able to interfere with matches.

[Racer26]

Originally Posted by Al Skierkiewicz

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

Al,

Nick was referring to the intentional interferer. How did THEY find the bug, and test it, before using it on Einstein.

Lots of people (including the report) don't buy the person's story that they "accidentally" found out by killing 2056 in SF2-1.

@Ed Law: I hadn't thought of the REASON they came to the FTA's, but now that you say that, I think you're right. They probably suspected that someone else knew, and was targeting THEIR alliance. Otherwise, why would they admit to the crime they'd (to that point, anyway) gotten away with?

[Nick Lawrence]

Originally Posted by Al Skierkiewicz

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

Al,

Sorry, I should have been a little more clear about what I meant. I understand this was an issue with the D-Link robot AP firmware. What I meant is when did the individual themselves find this issue, and when did he prove to himself the existence of this issue with the firmware.

I in no way meant for that to be intended as "Why didn't FIRST know about this earlier?" or something of that nature.

I also want to comment on the "witch hunt" that CD is seemingly brewing. While I would love to hear from this person about the hole itself and how they discovered it, I think they have already paid the price for it. I think for their personal identity to be revealed would only lead to more despair for that person. However, I would like to see a statement from their team over the subject. I think we'll see one soon.

-Nick

[techhelpbb]

Originally Posted by Nick Lawrence

Al,

Sorry, I should have been a little more clear about what I meant. I understand this was an issue with the D-Link robot AP firmware. What I meant is when did the individual themselves find this issue, and when did he prove to himself the existence of this issue with the firmware.

The problem is that other people might have also discovered it through independent means (as Al has acknowledged and I have mentioned word got around prior to Einstein that something might be a problem with the A version of the 1522 so they would have a place to start looking).

More importantly even if you find out this person is the tip of the iceberg, who is to say how straight an answer we'll get about the rest of the story.

Plus if we assume that someone was using it to cheat we'd have to assume they'd not be so silly to get caught otherwise it's of no value to cheat. This person knew that other people were aware of what they might be capable of. Logically all suspicion would flow to this person. It's not the best way to cheat and not get caught.

As others have written, maybe the individual feared the use of this attack against their interests (it doesn't matter either way with regards to their involvement).

With all the other issues I don't think we'll ever know the full extent of how much the 2 wireless issues could have contributed.

[rocknthehawk]

Originally Posted by Ed Law

I too waited a couple of days before I post. I read the letter, the report and every post on CD regarding this. I agree with Greg and others that the team and the individual should come out in the open and explain what happened. Let's not worry about other people's reaction when they find out. If they want to be angry, let them be angry. If they want to hold a grudge against the team forever, let them do that. If they want to forgive and move on, let them do that. We have no control on what other people think. It should not be a reason whether the individual and the team should come out in the open or not.

I am not making wild speculation here. Based on what I have read and putting my logic hat on, the individual seems to be one of the mentors of one of the Einstein teams. Based on the report and posts on CD, three of the four alliances can be cleared for various reasons. That leaves one alliance of 3 teams. Considering what happened in the semifinal round, there is motive why this individual may want to talk to the FTAs. This part is speculation so I am not going to name the team. I may be completely wrong here.

I think the team will come forward and do the right thing. I also don't think they should accept the free registration for next year. Let's give them some time. The report just came out over the weekend. They probably need to have a team meeting to discuss this and let everyone on the team know and draft a proper statement.

This has been my thinking as well. I don't think I could have worded it any better. My question is, was the Einstein incident the first and only time this person had purposely sabotaged a team?

[Al Skierkiewicz]

Originally Posted by techhelpbb

The problem is that other people might have also discovered it through independent means (as Al has acknowledged and I have mentioned word got around prior to Einstein that something might be a problem with the A version of the 1522 so they would have a place to start looking).

Brian,

I did not acknowledge that there is an issue with the A ver of the 1522. The problem is an interaction of the firmware loaded on the Cisco router to fix another problem noted earlier in the season when used with A version. It was discovered in testing after St. Louis that the Cisco firmware and the Dlink AP were affected when used together. Please read page 7 and 8 of the report. Please note that the report also states that this updated firmware was only installed at week four events. That being said, a user testing with wifi devices at their home field could not have discovered this vulnerability. In addition, it could only be discovered at an event using a 5GHz enabled wifi device of which, until recently, there were limited numbers of such devices available.

[techhelpbb]

Originally Posted by Al Skierkiewicz

Brian,

I did not acknowledge that there is an issue with the A ver of the 1522. The problem is an interaction of the firmware loaded on the Cisco router to fix another problem noted earlier in the season when used with A version. It was discovered in testing after St. Louis that the Cisco firmware and the Dlink AP were affected when used together. Please read page 7 and 8 of the report. Please note that the report also states that this updated firmware was only installed at week four events. That being said, a user testing with wifi devices at their home field could not have discovered this vulnerability. In addition, it could only be discovered at an event using a 5GHz enabled wifi device of which, until recently, there were limited numbers of such devices available.

I understand completely I read the report. The issue remains that I distinctly remember people telling me they had some unspecified issues with the 1522 version A weeks before Einstein. In fact I have 25's A version 1522 which they thought was intermittent and it's not and they replaced that before they arrived at the championships. That could mean that someone was messing with them or not (all depends on the other factors).

So this means that people had a place to start looking weeks before Einstein. Whether they could find the specific interaction as you said would require a Cisco unit with that firmware (and very few people knew that information). I suppose one could take from that they discovered this by messing with other fields (who knows when or how).

It seems from what I've seen that this person was sure they had something (which is sort of damning).

I was never able to find anything extremely unusual about this 1522 I got from them. So it fits that you'd need this and some external set of factors that do not exist in my environment but do exist on the actual field. Course I can only rely on the information I was told that nothing besides this AP was changed to remedy the issue (otherwise one could argue that something was an issue in their robot as additional stimulus and had changed by the time you reviewed it for this report).

The only way I can think of that someone could have stumbled on this without effecting a field would be to have exactly what's on the field. This seems unlikely given the way Cisco handles firmware and we know the version of firmware matters. That or perhaps they could have compared the 2 versions of the D-Link 1522 AP noticed a change in the behavior which points to the soft spot.

Obviously attacking the field till you find it would be easier and in this case require less technical skill and resources. Course that would almost certainly mean that whoever insisted on bringing this forward had already done something they shouldn't or seen someone else do it.

[Ether]

In the last paragraph of page 9 of the report, it states:

Over the course of these tests, FRC Engineering was able to determine how to identify a failed client authentication through the log data recorded in the field access point. However, the configuration of the field access points used during the 2012 FRC competitions, including the matches on Einstein, is such that log data is not retained when the access point is powered off.

Then on page 22 of the report:

Root Cause: Failed client authentication
Mitigation: Field Hardware Logging

Could someone with direct knowledge please confirm/clarify: Is the "Field Hardware Logging" mentioned on page 22 specifically referring to retention of the field access point logs mentioned on page 9?

Thank you.

[Al Skierkiewicz]

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

[techhelpbb]

Originally Posted by Al Skierkiewicz

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

I grant you this was hardly the only issue. However, those initial reports were enough for me to have both versions of the D-Link AP and compare them. I am persistent and that's a mere $200 to get 2 units to test (nothing I found pointed to the issue). Besides off season events do not get a spare parts kit. So the worst case was I had spare radios to offer in case something went down at the off season events. Besides I was testing my tiny oscilloscopes at off season events and they were on the power into the radio so if something had happened (and it didn't) I could offer a replacement as compensation. When I finished testing my oscilloscopes I merely gave some new 1522 AP away in trade for samples of misbehaving APs.

Obviously I didn't spend all that time and money to build those little oscilloscopes because I thought FIRST merely had AP issues (though I admit that while I knew and still know more ways someone could interfere with the wireless I never thought anyone would be that devious or in this case so easily caught).

Still it leaves it out in the open that at some point, perhaps multiple times, someone tested that exploit before they tried to demonstrate it. Additionally, I do agree with what you seem to conclude that this person didn't intend to rig the rankings. Surely this particular tampering is not the only issue and focusing on it too much distracts from the bigger issues we all must face from it. I'm not sure we'll ever know the full extent of what happened without adequate logs.

[Joon Park]

I also agree with what Greg and many others have said, that the individual should come out and admit their mistakes. I have full faith that the FIRST community at large will accept this individual's apology and move on, having learned from this ordeal.

But what I would not like to do is add on to this pressure that's amounting on this individual to speak out. Whether I think he/she should or not is a different matter; the decision is not mine to make. If the individual chooses to remain hidden, he/she may have to deal with the consequences of the discontent team members, and that's up to the individual.

That said,

Originally Posted by Lil' Lavery

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.

I think this is very important as well, to keep things in perspective. FIRST is a community unlike the public at large, with the distinction lying on its gracious professionalism. I have no doubt that the community will continue to uphold and cherish this unique characteristic through difficult events such as this.

[Joe Johnson]

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

[Astrokid248]

Originally Posted by Joe Johnson

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

Is all of the above okay? Cos the way I see it, there are four different systems. In terms of the goal of FRC, we're still doing well, if not better. That's what I call the Chairman's system. Also, the lengths FIRST went to after championships are quite impressive, in what I guess is the organization system. Then, we have the system comprised of the elements of game play, both on the field and in the pits, and all's that proves is how human we are. The Astros sucked once they got to the World Series in 2004, but nobody says that the system is broken (well, at least not for teams sucking at that high a level). Finally, there's the robot connection system, which is the only broken system out of the group. And that's the forest Lil Lav is talking about, with the tree being our unknown attacker. It's not the guy's intentions; it's his actions and how repeatable they might be, how much further a future attacker can go if we don't plan for these things in the future. And it's a tricky line to walk. Do we outright ban cellphones? If you do, all the teams who use social media from the field side or can't afford cameras beyond their phones are being denied a large portion of their team identity. But clearly letting everyone have access is also an issue. So do we fix it on FIRST's side? Make the routers inaccessible? Because that's got a whole bunch of other issues associated with it. So let's talk about those issues, and find a solution to the problem of a guy with a smartphone knocking out robot connections, instead of finding a guy we can yell at for weeks on end.

[JamesTerm]

Originally Posted by Joe Johnson

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.
Joe J.

I think I get it... let's break this down shall we:

"
I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
"

"everyone wants to talk about..." This is not true but can be interpreted as an exaggerated expression to bring their point across... Ok that's fine.


"A lot of people" ... this is not you... and really not some that may appear that way now with their posts. For example Greg's post... (I'm picking on him because of the timing of his post and because I have great respect for him)... the post was indeed fixed on one point (i.e. the trees). I should add I know the robowranglers like us (and many teams) will indeed study every piece of this report and make sure that we address each point properly.

So let's just say some people... when it seems like a lot of people.

The last part... missing the forest for the trees.

The reason why I'm going through the trouble of breaking this down is because I got this feeling as well, but it is just a feeling. Basically there are posts that are applying pressure to the perpetrator and/or team, (i.e. the trees)... the forest is all the other problems not talked about as much... I'd say these are the other technical responses some of which I've been trying to say. I know someone else mentioned the idea of splitting the discussion up as well.

I know when the dust settles I will want to discuss some of the other aspects of the report such as the network capping. Right now... I feel like backing off and letting the predominant discussion carry on.

[Steven Donow]

Originally Posted by JamesTerm

I think I get it... let's break this down shall we:

"
I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
"

"everyone wants to talk about..." This is not true but can be interpreted as an exaggerated expression to bring their point across... Ok that's fine.


"A lot of people" ... this is not you... and really not some that may appear that way now with their posts. For example Greg's post... (I'm picking on him because of the timing of his post and because I have great respect for him)... the post was indeed fixed on one point (i.e. the trees). I should add I know the robowranglers like us (and many teams) will indeed study every piece of this report and make sure that we address each point properly.

So let's just say some people... when it seems like a lot of people.

The last part... missing the forest for the trees.

The reason why I'm going through the trouble of breaking this down is because I got this feeling as well, but it is just a feeling. Basically there are posts that are applying pressure to the perpetrator and/or team, (i.e. the trees)... the forest is all the other problems not talked about as much... I'd say these are the other technical responses some of which I've been trying to say. I know someone else mentioned the idea of splitting the discussion up as well.

I know when the dust settles I willwant to discuss some of the other aspects of the report such as the network capping. Right now... I feel like backing off and letting the predominant discussion carry on.

So based off your interpretation... We as a general FIRST community need to focus on the forest.... Let the specific individuals affected focus on the trees. These are my feelings at least.