[FRC Blog] Einstein Report Released

[Nick Lawrence]

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

[Camren]

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

[techhelpbb]

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

As I've pointed out before in this topic. It's possible that the power supply that feeds the AP comes up in such a way that causes the router to boot in a state that is not useful.

It's the sort of thing that would depend on what the status of the charge is in the capacitors in the system when you power up. Also it would depend on having a AP a little more sensitive than the median.

I tested a few robots at off season events with tiny oscilloscopes attached to them on the field. Sometimes during a power up it takes a little longer to get to regulation voltage than at other times (we're talking milliseconds max here not seconds).

So it's hard to say that every time you had to reboot an AP it was locked up by a bad processor reset or the exploit. Unfortunately as others have pointed out there are good indicators of the exploit behavior but it wasn't tracked throughout the season.

Hard to say how much was interloper, how much was component malfunction and how much build related failure.

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

This has come up over and over.

If one assumes that more than one person was aware of the exploit.

There is no assurance that the other people that know are on the team associated with the individual.

In fact the deauth attack (there are 2 issues in the report) could easily have been exploited by anyone anywhere. All you'd have to do is Google it.

So no it's not logical to assume the team had to have known or was the only possible exploiter.

More importantly the individual didn't need to do anything really all that unusual besides be too aggressive attempting to connect to the network.

That's too easy for anyone, even a spectator, to do in mere curiosity.

Once we assume that more than just this individual might be involved who is to say that we can trust that someone somewhere won't do something unfortunate to the individual as it's really the same problem:

1. Everyone who keeps trying to find out is doing something FIRST may not be comfortable with.
2. When someone does find out we don't know how they'll behave.
3. Once the cat is out of the bag we won't know the extent or duration of the consequences metered out to this person.

By actively seeking out this person we are very much doing what we all indicated is bad behavior on their part.

Worse as a community (just like this person's team) we'd take the hit for anyone that went overboard.

We're setting ourselves up for a vicious cycle.

[Chris Fultz]

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure.

Greg

For clarity, there were multiple root causes identified - some hardware, some software, some robot systems, some unknown, and intentional interference.

[Al Skierkiewicz]

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein?

-Nick

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

[Chinmay]

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

I think what Nick might of meant, and what I personally am thinking about isn't when FIRST discovered the bug, but when the individual discovered that he/she was able to interfere with matches.

[Racer26]

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

Al,

Nick was referring to the intentional interferer. How did THEY find the bug, and test it, before using it on Einstein.

Lots of people (including the report) don't buy the person's story that they "accidentally" found out by killing 2056 in SF2-1.

@Ed Law: I hadn't thought of the REASON they came to the FTA's, but now that you say that, I think you're right. They probably suspected that someone else knew, and was targeting THEIR alliance. Otherwise, why would they admit to the crime they'd (to that point, anyway) gotten away with?

[Nick Lawrence]

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

Al,

Sorry, I should have been a little more clear about what I meant. I understand this was an issue with the D-Link robot AP firmware. What I meant is when did the individual themselves find this issue, and when did he prove to himself the existence of this issue with the firmware.

I in no way meant for that to be intended as "Why didn't FIRST know about this earlier?" or something of that nature.

I also want to comment on the "witch hunt" that CD is seemingly brewing. While I would love to hear from this person about the hole itself and how they discovered it, I think they have already paid the price for it. I think for their personal identity to be revealed would only lead to more despair for that person. However, I would like to see a statement from their team over the subject. I think we'll see one soon.

-Nick

[techhelpbb]

Al,

Sorry, I should have been a little more clear about what I meant. I understand this was an issue with the D-Link robot AP firmware. What I meant is when did the individual themselves find this issue, and when did he prove to himself the existence of this issue with the firmware.


The problem is that other people might have also discovered it through independent means (as Al has acknowledged and I have mentioned word got around prior to Einstein that something might be a problem with the A version of the 1522 so they would have a place to start looking).

More importantly even if you find out this person is the tip of the iceberg, who is to say how straight an answer we'll get about the rest of the story.

Plus if we assume that someone was using it to cheat we'd have to assume they'd not be so silly to get caught otherwise it's of no value to cheat. This person knew that other people were aware of what they might be capable of. Logically all suspicion would flow to this person. It's not the best way to cheat and not get caught.

As others have written, maybe the individual feared the use of this attack against their interests (it doesn't matter either way with regards to their involvement).

With all the other issues I don't think we'll ever know the full extent of how much the 2 wireless issues could have contributed.

[rocknthehawk]

I too waited a couple of days before I post. I read the letter, the report and every post on CD regarding this. I agree with Greg and others that the team and the individual should come out in the open and explain what happened. Let's not worry about other people's reaction when they find out. If they want to be angry, let them be angry. If they want to hold a grudge against the team forever, let them do that. If they want to forgive and move on, let them do that. We have no control on what other people think. It should not be a reason whether the individual and the team should come out in the open or not.

I am not making wild speculation here. Based on what I have read and putting my logic hat on, the individual seems to be one of the mentors of one of the Einstein teams. Based on the report and posts on CD, three of the four alliances can be cleared for various reasons. That leaves one alliance of 3 teams. Considering what happened in the semifinal round, there is motive why this individual may want to talk to the FTAs. This part is speculation so I am not going to name the team. I may be completely wrong here.

I think the team will come forward and do the right thing. I also don't think they should accept the free registration for next year. Let's give them some time. The report just came out over the weekend. They probably need to have a team meeting to discuss this and let everyone on the team know and draft a proper statement.

This has been my thinking as well. I don't think I could have worded it any better. My question is, was the Einstein incident the first and only time this person had purposely sabotaged a team?

[Al Skierkiewicz]

The problem is that other people might have also discovered it through independent means (as Al has acknowledged and I have mentioned word got around prior to Einstein that something might be a problem with the A version of the 1522 so they would have a place to start looking).

Brian,

I did not acknowledge that there is an issue with the A ver of the 1522. The problem is an interaction of the firmware loaded on the Cisco router to fix another problem noted earlier in the season when used with A version. It was discovered in testing after St. Louis that the Cisco firmware and the Dlink AP were affected when used together. Please read page 7 and 8 of the report. Please note that the report also states that this updated firmware was only installed at week four events. That being said, a user testing with wifi devices at their home field could not have discovered this vulnerability. In addition, it could only be discovered at an event using a 5GHz enabled wifi device of which, until recently, there were limited numbers of such devices available.

[techhelpbb]

Brian,

I did not acknowledge that there is an issue with the A ver of the 1522. The problem is an interaction of the firmware loaded on the Cisco router to fix another problem noted earlier in the season when used with A version. It was discovered in testing after St. Louis that the Cisco firmware and the Dlink AP were affected when used together. Please read page 7 and 8 of the report. Please note that the report also states that this updated firmware was only installed at week four events. That being said, a user testing with wifi devices at their home field could not have discovered this vulnerability. In addition, it could only be discovered at an event using a 5GHz enabled wifi device of which, until recently, there were limited numbers of such devices available.

I understand completely I read the report. The issue remains that I distinctly remember people telling me they had some unspecified issues with the 1522 version A weeks before Einstein. In fact I have 25's A version 1522 which they thought was intermittent and it's not and they replaced that before they arrived at the championships. That could mean that someone was messing with them or not (all depends on the other factors).

So this means that people had a place to start looking weeks before Einstein. Whether they could find the specific interaction as you said would require a Cisco unit with that firmware (and very few people knew that information). I suppose one could take from that they discovered this by messing with other fields (who knows when or how).

It seems from what I've seen that this person was sure they had something (which is sort of damning).

I was never able to find anything extremely unusual about this 1522 I got from them. So it fits that you'd need this and some external set of factors that do not exist in my environment but do exist on the actual field. Course I can only rely on the information I was told that nothing besides this AP was changed to remedy the issue (otherwise one could argue that something was an issue in their robot as additional stimulus and had changed by the time you reviewed it for this report).

The only way I can think of that someone could have stumbled on this without effecting a field would be to have exactly what's on the field. This seems unlikely given the way Cisco handles firmware and we know the version of firmware matters. That or perhaps they could have compared the 2 versions of the D-Link 1522 AP noticed a change in the behavior which points to the soft spot.

Obviously attacking the field till you find it would be easier and in this case require less technical skill and resources. Course that would almost certainly mean that whoever insisted on bringing this forward had already done something they shouldn't or seen someone else do it.

[Ether]

In the last paragraph of page 9 of the report, it states:

Over the course of these tests, FRC Engineering was able to determine how to identify a failed client authentication through the log data recorded in the field access point. However, the configuration of the field access points used during the 2012 FRC competitions, including the matches on Einstein, is such that log data is not retained when the access point is powered off.

Then on page 22 of the report:

Root Cause: Failed client authentication
Mitigation: Field Hardware Logging

Could someone with direct knowledge please confirm/clarify: Is the "Field Hardware Logging" mentioned on page 22 specifically referring to retention of the field access point logs mentioned on page 9?

Thank you.

[Al Skierkiewicz]

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

[techhelpbb]

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

I grant you this was hardly the only issue. However, those initial reports were enough for me to have both versions of the D-Link AP and compare them. I am persistent and that's a mere $200 to get 2 units to test (nothing I found pointed to the issue). Besides off season events do not get a spare parts kit. So the worst case was I had spare radios to offer in case something went down at the off season events. Besides I was testing my tiny oscilloscopes at off season events and they were on the power into the radio so if something had happened (and it didn't) I could offer a replacement as compensation. When I finished testing my oscilloscopes I merely gave some new 1522 AP away in trade for samples of misbehaving APs.

Obviously I didn't spend all that time and money to build those little oscilloscopes because I thought FIRST merely had AP issues (though I admit that while I knew and still know more ways someone could interfere with the wireless I never thought anyone would be that devious or in this case so easily caught).

Still it leaves it out in the open that at some point, perhaps multiple times, someone tested that exploit before they tried to demonstrate it. Additionally, I do agree with what you seem to conclude that this person didn't intend to rig the rankings. Surely this particular tampering is not the only issue and focusing on it too much distracts from the bigger issues we all must face from it. I'm not sure we'll ever know the full extent of what happened without adequate logs.

[Joon Park]

I also agree with what Greg and many others have said, that the individual should come out and admit their mistakes. I have full faith that the FIRST community at large will accept this individual's apology and move on, having learned from this ordeal.

But what I would not like to do is add on to this pressure that's amounting on this individual to speak out. Whether I think he/she should or not is a different matter; the decision is not mine to make. If the individual chooses to remain hidden, he/she may have to deal with the consequences of the discontent team members, and that's up to the individual.

That said,

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
I think this is very important as well, to keep things in perspective. FIRST is a community unlike the public at large, with the distinction lying on its gracious professionalism. I have no doubt that the community will continue to uphold and cherish this unique characteristic through difficult events such as this.

[Joe Johnson]

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

[Astrokid248]

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

Is all of the above okay? Cos the way I see it, there are four different systems. In terms of the goal of FRC, we're still doing well, if not better. That's what I call the Chairman's system. Also, the lengths FIRST went to after championships are quite impressive, in what I guess is the organization system. Then, we have the system comprised of the elements of game play, both on the field and in the pits, and all's that proves is how human we are. The Astros sucked once they got to the World Series in 2004, but nobody says that the system is broken (well, at least not for teams sucking at that high a level). Finally, there's the robot connection system, which is the only broken system out of the group. And that's the forest Lil Lav is talking about, with the tree being our unknown attacker. It's not the guy's intentions; it's his actions and how repeatable they might be, how much further a future attacker can go if we don't plan for these things in the future. And it's a tricky line to walk. Do we outright ban cellphones? If you do, all the teams who use social media from the field side or can't afford cameras beyond their phones are being denied a large portion of their team identity. But clearly letting everyone have access is also an issue. So do we fix it on FIRST's side? Make the routers inaccessible? Because that's got a whole bunch of other issues associated with it. So let's talk about those issues, and find a solution to the problem of a guy with a smartphone knocking out robot connections, instead of finding a guy we can yell at for weeks on end.

[JamesTerm]

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.
Joe J.


I think I get it... let's break this down shall we:

"
I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
"

"everyone wants to talk about..." This is not true but can be interpreted as an exaggerated expression to bring their point across... Ok that's fine.


"A lot of people" ... this is not you... and really not some that may appear that way now with their posts. For example Greg's post... (I'm picking on him because of the timing of his post and because I have great respect for him)... the post was indeed fixed on one point (i.e. the trees). I should add I know the robowranglers like us (and many teams) will indeed study every piece of this report and make sure that we address each point properly.

So let's just say some people... when it seems like a lot of people.

The last part... missing the forest for the trees.

The reason why I'm going through the trouble of breaking this down is because I got this feeling as well, but it is just a feeling. Basically there are posts that are applying pressure to the perpetrator and/or team, (i.e. the trees)... the forest is all the other problems not talked about as much... I'd say these are the other technical responses some of which I've been trying to say. I know someone else mentioned the idea of splitting the discussion up as well.

I know when the dust settles I will want to discuss some of the other aspects of the report such as the network capping. Right now... I feel like backing off and letting the predominant discussion carry on.

[Steven Donow]

I think I get it... let's break this down shall we:

"
I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
"

"everyone wants to talk about..." This is not true but can be interpreted as an exaggerated expression to bring their point across... Ok that's fine.


"A lot of people" ... this is not you... and really not some that may appear that way now with their posts. For example Greg's post... (I'm picking on him because of the timing of his post and because I have great respect for him)... the post was indeed fixed on one point (i.e. the trees). I should add I know the robowranglers like us (and many teams) will indeed study every piece of this report and make sure that we address each point properly.

So let's just say some people... when it seems like a lot of people.

The last part... missing the forest for the trees.

The reason why I'm going through the trouble of breaking this down is because I got this feeling as well, but it is just a feeling. Basically there are posts that are applying pressure to the perpetrator and/or team, (i.e. the trees)... the forest is all the other problems not talked about as much... I'd say these are the other technical responses some of which I've been trying to say. I know someone else mentioned the idea of splitting the discussion up as well.

I know when the dust settles I willwant to discuss some of the other aspects of the report such as the network capping. Right now... I feel like backing off and letting the predominant discussion carry on.
So based off your interpretation... We as a general FIRST community need to focus on the forest.... Let the specific individuals affected focus on the trees. These are my feelings at least.

[CalTran]

I can't help but feel like this report is somewhat backfiring on FIRST. While I do agree that it has been a vital process and do applaud them for releasing every detail about it, it appears to have created a rift in the FIRST community; splitting people into people who want to move past this at god speed, and a group who, and rightfully so, still attempting to cope with what they've faced. This second group, as exemplified by posts from some of the most esteemed members of the community, is a group most of us will never understand, and hopefully never will have to understand. A sweep of both Worlds and a CCA is, or at least should, be every teams goal, and a once in a lifetime event. To have been so close, and have everything seem to have been ripped away by a single person would be devastating to me. I would never be able to find it in my heart to forgive that individual.

I guess what I'm trying to say is that these teams will need time to cope with what happened, and that the rest of the FIRST community should give them ample time to heal at their own pace. When they're ready to forgive and maybe even forget, that is the time we can all look back at this.

[techhelpbb]

I can't help but feel like this report is somewhat backfiring on FIRST. While I do agree that it has been a vital process and do applaud them for releasing every detail about it, it appears to have created a rift in the FIRST community; splitting people into people who want to move past this at god speed, and a group who, and rightfully so, still attempting to cope with what they've faced. This second group, as exemplified by posts from some of the most esteemed members of the community, is a group most of us will never understand, and hopefully never will have to understand. A sweep of both Worlds and a CCA is, or at least should, be every teams goal, and a once in a lifetime event. To have been so close, and have everything seem to have been ripped away by a single person would be devastating to me. I would never be able to find it in my heart to forgive that individual.

I guess what I'm trying to say is that these teams will need time to cope with what happened, and that the rest of the FIRST community should give them ample time to heal at their own pace. When they're ready to forgive and maybe even forget, that is the time we can all look back at this.

I have a lot of goals with FIRST, but sweeping the competition merely in search of being the highest scoring team over and over is rarely one of them. It's welcome when it happens but I can't loose sight that this is about overcoming adversity. Those other awards at the end are not merely second, third, forth, etc.

I'm not sure how discussing moving forward harms the people that experienced this. There's nothing that can be done to undo this situation. Perhaps award them additional championship winners but that's beyond my ability to offer.

We do not have the logs to determine how far back that particular individual's attack vector was actually used (or whether they were the only exploiter of it). So the other people that almost certainly were denied beyond the Einstein teams will never know how or why.

We do not know if or when anyone used a deauth attack vector anywhere because again there are no logs.

We do not know how many times a robot made it to a championship or even highly ranked with a technical problem that might be caught with more frequent examination, better tools, or more time to look. How many of those teams were denied on the premise that the top level teams are more effectively mitigating those issues? Only to discover these issues remain at the top of the ranks. No disrespect can be placed on any team as this has been the nature of the competition for a good long time. Equal playing field and equal expectation of common issues.

I'm not rushing them, and I don't think anyone else is either, to forget their pain that would be totally unacceptable. I'm merely pointing out that their pain can't stop the world and their pain can't be a good reason to ignore the direction FIRST has clearly chosen for handling the identity of this team or it's members.

This is a tragic consequence, but the pain of tragedy is often the crushingly simple fact that you carry on and no matter how many times you talk about it will still be a tragedy.

If anyone can find a more suitable memorial for this I'll be happy to contribute but not at the expense of loosing the value of the report or subjecting future teams to a closely related situation.

[3747Mentor]

I think it is ok to be angry right now. I think it is ok to be straight livid. I think it is ok to be hurt. I think we can be mad for the Einstein teams and for ourselves. We all have skin in this game. We are coaches, players, supporters, and fans. Imagine the outrage if it was found out the Super Bowl was rigged. There would be congressional investigations. It would blow up the 24 hour news cycle. There may even be riots.

Well, this was our Super Bowl, and someone fixed it! Someone tampered with our biggest stage and there are calls to laugh it off and look at the bright side. It is way too soon to be upset with someone for being angry. I was not on Einstein and I am angry! I am angry for those who were immediately affected by this despicable act and I am angry for my team as well. We are competitors. We want to know that our hard work and effort goes into a competition that is truly played on a level field. This person took that away and I am not ready to get over it. There are still too many unanswered questions that need to be resolved. I want to know the whole story, and right now, only a handful of people do.

Einstein is not only important for those involved, but it is extremely important to all of us as well. I read that it is not about the robots, or even about winning, but I disagree. The robots are the vehicle to inspiration about STEM. Winning increases the impact. It is the motivation to improve ourselves. Professionals need to win contracts so they must find the best/most cost-effective solutions. Our goal is to win a competition. We don’t build a robot to do show and tell (the time for that is after the competition season is over) we build one to win Einstein. Competition drives us to improve. We can’t all win Einstein, so when we don’t, we look toward the teams that do make it to serve as our inspiration and motivation for improvement next season. We learn things from the teams involved and try them in the offseason. We use their excellence to make ourselves better. Those teams were cheated from an honest competition, and so were we.

I can see trouble in this thread for us. We are fighting amongst ourselves about what teams should be satisfied with and when they should be ready to move on. Arguments are getting personal and off the subject at hand. Are we really angry at each other, or are we angry at the person who lit this candle. I think this community might be better served by expressing our outrage at the person who did this..by airing it out. I see in this thread and the Sabotage thread that was just closed that we are starting to turn it against ourselves, and I believe it is all misplaced anger. Who are we really angry at?

I am a pilot, and I know from experience that a great flight can be ruined in the eyes of passengers with a bad landing. Bounce one on hard and passengers talk about the whole experience as if it was horrible. Unfortunately, for FRC, Einstein was 2012’s landing. Only time will let the season be put into perspective. Please stop telling people to look at the positives. They will later. If you are ready, good, we need people looking forward, but don’t rush the others. These arguments will only further hurt us, which is exactly what we don’t need right now.

[cgmv123]

See other thread: Einstein report powerpoint (http://www.chiefdelphi.com/forums/showthread.php?t=107350)

[techhelpbb]

I can see trouble in this thread for us. We are fighting amongst ourselves about what teams should be satisfied with and when they should be ready to move on. Arguments are getting personal and off the subject at hand. Are we really angry at each other, or are we angry at the person who lit this candle. I think this community might be better served by expressing our outrage at the person who did this..by airing it out. I see in this thread and the Sabotage thread that was just closed that we are starting to turn it against ourselves, and I believe it is all misplaced anger. Who are we really angry at?


I am sure that most of us are not angry at anyone. I surely am not angry. However, I am concerned that when winning becomes the only acceptable end that the value of the journey is lost. The simple reality is most of us will not win the championship in any given year.

Yes we do aspire to win. We do hope and we do the best we can to win.

Then again many of us will stop like these Einstein teams did and help each other out when it's not in our interest to make sure we win.

Unfortunately, there's clearly a much larger element of chance at work here than we seem to be able to accept as a community. I continue to see people speak of the difficulty of the best of the best teams to reach the highest echelon of competition.

I have been publicly quite pointed about the electrical issues that could impact robots since long before Einstein and long before this report. This report makes it clear that the best of our best still have problems that we've often assumed should not exist at that level of competition.

The combination of problems like the electrical issues from this report and the random uncertainty of the game designs themselves clearly makes it unpredictable that the qualities we think frame the best of the best are any assurance that they'll succeed. Hence the other awards.

The actions of this interloper aside. The uncertainty added to this environment beyond the game design is a fundamental problem that makes this all the more devastating for those most directly impacted. Surely it's an issue that adds more salt to these teams' wounds than is necessary. Even if they do everything just the same as they did this year there's no way they can be sure they'll place at that level again. FIRST's offer to promote them up to at least the venue automatically may only be slight help to them.

To use your example, we hope in the STEM fields when we compete on cost, quality and price we compete on a fair playing field. In the real world we often also compete against politics and tactics that exceed honest business.

I see a lot of frustration in this community which holds STEM values so dear to themselves that we have these uncertainties and lack of logs to find the proofs we so value. I do not think it'll turn to anger or hatred at random. Still I wouldn't fuel that fire with jokes because right now I'm sure some people still need the uneasy peace.

Clearly work needs to be done to limit the random impacts on the game play to those aspects which give opportunity for benefit to those that exhibit the traits we as a competition hold as the best example.

[Sean Raia]

It is rather sad, but it is my belief that this person will not reveal themself.

Anyone who is willing to interfere with robot communications to try and give their team the advantage is likely a coward. Cowards dont do whats right, they do whats easy.
Its fair to be upset with them...

FIRST has done an excellent job documenting their investigation and for that I thank them.

[KrazyCarl92]

Unfortunately, we've seen some of the most disrespectful posts ever in these forums directed at the victims of this incident. Although I can't claim to understand what the members of these teams are going through, I know for a fact that I would be distraught and disheartened for a while following the results of the Investigation if it were me. The last thing these teams need right now is for members of this community to be insensitive to their situation. They are some of the greatest teams comprised of some of the best people that FIRST has to offer.

I know if my hard work was ruined by someone else's wrong doing, I might question the time, effort, and commitment that was invested in it. Even though it wasn't my work that was destroyed, I am still upset, angry, and ashamed that someone in our community would do such a thing. Matters are made worse by the disrespect shown to these teams by their own community. The last thing we need is for some of the best people in our community to question the time, effort, and commitment they devote to FIRST. I fear that if disrespect continues, it could cause even more permanent damage.

Let us unite with support for those effected by the events on Einstein. Please be understanding and allow them ample time to recover from the tragedy, however long that may be.

As for the perpetrator, rather than vilify him/her, an attempt to understand what led him/her to do this would be more appropriate. As Dean Kamen says, "Society gets what it celebrates." What we got on Einstein was an unfair attack. While I know no one celebrates unfair attacks, there had to be something mistakenly celebrated to lead to this result. Speculatively, I would have to say competing at all costs was celebrated where it would be more appropriate to celebrate competing with gracious professionalism. We can't know for sure, so understanding the thoughts behind the actions will serve as a valuable resource to prevent these attempts in the future.

Thank you FIRST, the Einstein competitors, and our volunteers for your thorough investigation of the incident. It makes me proud to be a part of an organization that is willing to go to such lengths for the benefit of our community.

[Camren]

Quick not I'm sure someone has already brought this up but as a gamer I have made my skin thicker against hackers so that when I hear foul play was suspected I can positively say "then up your game". Is the FMS the best way to run robots I cant say it is because if someone hacks that box they can do what ever they want, they could change the score by adding a few penalties in or seize communication on a bot or even a whole alliance So what is the best system? I don't know. My guess would be something where we didn't have to rely on every single aspect of the game in one software. But that's where you get the "Then up your game" from.

[Sean Raia]

Quick not I'm sure someone has already brought this up but as a gamer I have made my skin thicker against hackers so that when I hear foul play was suspected I can positively say "then up your game". Is the FMS the best way to run robots I cant say it is because if someone hacks that box they can do what ever they want, they could change the score by adding a few penalties in or seize communication on a bot or even a whole alliance So what is the best system? I don't know. My guess would be something where we didn't have to rely on every single aspect of the game in one software. But that's where you get the "Then up your game" from.

Reading this made me dizzy. Im sorry but id understand you better if you were to restructure that thought into something more coherent.
You cant relate this to video games, they are totally different in scale and impact.

[Steven Donow]

Reading this made me dizzy. Im sorry but id understand you better if you were to restructure that thought into something more coherent.
You cant relate this to video games, they are totally different in scale and impact.

I think he's saying that the FMS is one thing controlling everything (scoring, connection, etc...) and that it might be better to have everything be a separate system...but wouldn't all the systems integrated together just be the FMS!??!?! :confused:

Anyway, I think with what he's suggesting we wouldn't be able to have all the stuff we have now with the FMS twitter feed and whatnot.

[Ether]

...FMS twitter feed...

This is slightly off-topic but still generally related...

I'm looking for an open channel to someone at FIRST who would be sympathetic to the suggestion that the Twitter data be logged locally to non-volatile storage so it could be made available after-the-fact in those cases where the feed is blocked (http://www.chiefdelphi.com/forums/showpost.php?p=1147482&postcount=15) at the event.

There are amateur statisticians and historians in the community who would love to have complete data for analysis.

[Greg McKaskle]

I don't believe the report says anything about the FMS being hacked, scores being changed, or robots being seized. There is no evidence those actions occurred on Einstein.

The report discusses how the FIRST staff performed some typical DoS attacks on the bridge and router to learn what the symptoms would look like. The report discusses that a bug was discovered in the field wifi components that allowed for a disruption of service. The FIRST staff then explored the various symptoms and the requirements for the bug to manifest. The bug allowed for service disruptions, but no foreign device joined any field access point.

Also, the exploit required no hacking skills. Hackers everywhere are cringing when this is referred to as a hack. The term hack never appears in the report. Sorry to be such a stickler for terminology, but inaccurate descriptions of what took place do not help matters. If there are parts of the report which need clarification, please ask rather than jump to conclusions.

Greg McKaskle

[Steven Donow]

I don't believe the report says anything about the FMS being hacked, scores being changed, or robots being seized. There is no evidence those actions occurred on Einstein.

The report discusses how the FIRST staff performed some typical DoS attacks on the bridge and router to learn what the symptoms would look like. The report discusses that a bug was discovered in the field wifi components that allowed for a disruption of service. The FIRST staff then explored the various symptoms and the requirements for the bug to manifest. The bug allowed for service disruptions, but no foreign device joined any field access point.

Also, the exploit required no hacking skills. Hackers everywhere are cringing when this is referred to as a hack. The term hack never appears in the report. Sorry to be such a stickler for terminology, but inaccurate descriptions of what took place do not help matters. If there are parts of the report which need clarification, please ask rather than jump to conclusions.

Greg McKaskle

The FMS was hacked in the way that 14 year old's Facebooks are "hacked" by their best friend who posted, "I smell" while they are both in the same room.

(I've been waiting for someone to post something like that so I can make the joke :o )

[techhelpbb]

I don't believe the report says anything about the FMS being hacked, scores being changed, or robots being seized. There is no evidence those actions occurred on Einstein.

The report discusses how the FIRST staff performed some typical DoS attacks on the bridge and router to learn what the symptoms would look like. The report discusses that a bug was discovered in the field wifi components that allowed for a disruption of service. The FIRST staff then explored the various symptoms and the requirements for the bug to manifest. The bug allowed for service disruptions, but no foreign device joined any field access point.

Also, the exploit required no hacking skills. Hackers everywhere are cringing when this is referred to as a hack. The term hack never appears in the report. Sorry to be such a stickler for terminology, but inaccurate descriptions of what took place do not help matters. If there are parts of the report which need clarification, please ask rather than jump to conclusions.

Greg McKaskle

There are 2 vectors in that report.

The confirmed vector was the one that needed very little beyond a phone. They found someone that admitted to that on Einstein.

The other vector (which does work but we have no evidence either way it was used) was deauth and generally that one is described on hack a day in October 2011.

As a person that works in computer security I know most big bad 'hackers' people find are just exploiting the much more time consuming efforts of others.

In this person's case it is more social engineering. They must have tested this before they reported it. The manipulation is in reporting it in such a way we will not be able to find out how and when that was done before.

Course they may not have realized that there were insufficient logs stored on the field servers so that was a gamble.

Still there is no evidence presented to support the idea that this person intended to influence the Einstein matches in a particular direction (who got hit was just a function of proving it worked at all).

The trick with the phone wouldn't have worked on at least 2 of the robots because those 2 had the B version of the D-Link AP on them at the time. Without logs we have no way of knowing whether the person with the phone knew that those 2 teams had that B version AP and ignored them in their effort. So there's no reason to suspect that this person knew anything more than this trick they pulled worked before somewhere, somehow.

Keeping in mind that this trick with the phone also requires the Cisco field AP to have a specific version of firmware the only practical place to test that without heavy reverse engineering would be on a field.

[Greg McKaskle]

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle

[Billfred]

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle
This. This. A million times, this.

If you're clever enough to figure out things like this, you should be smart enough to realize you have a duty to keep this knowledge out of the wrong hands and disclose it properly.

[JamesTerm]

The trick with the phone wouldn't have worked on at least 2 of the robots because those 2 had the B version of the D-Link AP on them at the time.


Ah HA! Those two robots must have been on the Blue alliance!
;)

[techhelpbb]

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle

In the report it says this:

Page 10:

"While the Einstein matches were in progress, an individual was observed near the field using a cell phone in an apparent attempt to access the field WiFi network. This individual had attempted to engage field personnel in discussions while the field personnel were troubleshooting other issues. This individual was asked to put away the cell phone, and complied. Later, the individual was observed using the cell phone again, and at that point, before the last two Einstein matches were played, was asked to leave the field area, and did so."

"After Championship, this individual came forward wishing to share knowledge regarding the failed client authentication issue. The individual claimed to have attempted to connect to the network associated with Team 2056 during Semi-Final 2-1 and observed that this attempt corresponded with the robot losing communication."

Posts were made with slightly more information. That was pages ago and I'll leave that to someone else to cull.

I would have to suspect that the attempt to engage field personnel was their initial attempt to report the issue.

This. This. A million times, this.

If you're clever enough to figure out things like this, you should be smart enough to realize you have a duty to keep this knowledge out of the wrong hands and disclose it properly.

Please be aware that there are currently several vectors I could shut down: all the robots, some of the robots, or halt any of your robots at will.

Remember I do computer security and it's part of my job and no one asks me my opinion about this at FIRST.

Additionally I know other people have reported vectors some of which are not listed and have yet to get a clear line to FIRST either.

Your assumption presumes that you can get the ear of the person you need to talk to and that in reporting it someone decides to mitigate it.

So for example perhaps someone tried to be the 'white hat' when they had a person face to face at Einstein and didn't like what they got for it (I presume only...it could also be that they were just trying to cozy up to field personnel to get better signal strength).

Ah HA! Those two robot must have been on the Blue alliance!
;)

Appendix A of the report is a little tricky to read because of how it's arranged in the left column.

Teams 16, 25, and 207 had the B version and would not have been effected by the stunt with the phone trying to connect.

[Greg McKaskle]

I'm sorry for paraphrasing, but at least one of the field personnel was an NI employee acting as CSA. No, it wasn't me, but I've heard this story from several points of view. I have to believe that there are better ways to gain signal strength. If you have access to Bill Miller, Frank Merrick, the FTAs, and virtually all of the FIRST staff wearing red ball caps, I think you can increase signal strength in many ways.

Greg McKaskle

[Denise Bohnsack]

It appears Marty McFly's Time Machine/Delorean would be useful now. It would be truly wonderful to go back and change these events and fix them before they had a chance to happen. :) Instead we are left with trying to prevent them in the future. I will leave this to the programmers and engineers.
I have followed this thread all week. For good reason, there is much anger and those effected have every right to feel this way for many reasons. As more than 1 poster has stated, this was a chance to have a blue flag for once. Just an idea to consider, may be a really bad one, but I will put it out there anyway just from a Mom's perspective. Could a flag/banner be created for this year's finalists that is extra special/limited edition/special color variety? Given to all that were in the finals since each might have had a chance at the title. Sort of a badge/banner of honor, for surviving and enduring the 2012 catastrophe? As the years go by, it would be special as no other team will have the chance to have such a banner, and the teams that go on can take pride if the fact they persevered and moved forward. I am in no way suggesting this makes up for what happened or replaces a title championship. I am suggesting this pays tribute to the teams who have endured much and are to be respected.
It is a small gesture but sometimes small gestures are very meaningful. I understand there would be an extra cost issue here which would need to be addressed to provide these teams with these banners, but I bet many of us who feel for these teams would gladly help. Again, just a Mom's suggestion and if some of you find it insulting or inadequate please forgive me, I promise I mean well. :) The only way to ever make a positive out of a very bad situation is to learn from it or make it better for those coming along next. Wish I had the perfect words or solution, but nobody does. FIRST will continue and let's try to make it better for future FIRSTers.

[techhelpbb]

I'm sorry for paraphrasing, but at least one of the field personnel was an NI employee acting as CSA. No, it wasn't me, but I've heard this story from several points of view. I have to believe that there are better ways to gain signal strength. If you have access to Bill Miller, Frank Merrick, the FTAs, and virtually all of the FIRST staff wearing red ball caps, I think you can increase signal strength in many ways.

Greg McKaskle

If in this case you equate 'signal strength' with finding someone who will listen then the question is how in someone's face can you get during a crisis (there were other issues at work). I don't know the particulars of the venue that day. I was at work in NYC watching over the Internet.

In this case I meant literal signal strength. Their ability to connect to the network over the any possible interference from other people would be improved by proximity to the field. Something you might be able to get by giving yourself a way to get close to it (say by talking to field personnel).

No matter what if they attempted to report it at all it would mean they tried it before that or saw someone else do it.

More importantly some of these attacks mirror the field troubles we noticed at Monty Madness off season the year before.
That was another example where I personally checked the D-Link AP and the robot and there were no problems.
Go out on the field and poof...

I did mention that in the topic about alternate control systems where you explained about the new logging features before the start of all the 2012 events.

I suspect this has been brewing for a while.

[Andy Baker]

Kudos to FIRST for the attention to detail, thoughtfulness, and completion of this report. The effort of FIRST, the volunteers and suppliers, and the Einstein 12 is much appreciated.

I really can't understand quite how the teams on Einstein feel about all of this, but I can understand the frustration, bewilderment, and anger. While the investigation is going to make things better, this situation is still tough to handle and is very frustrating. The sabotage was not just to certain teams, but to the entire FRC program and community.

Everyone builds their robots uniquely, runs their teams differently, and prefers all kinds of flavors of ice cream. Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Andy B.

[Barry Bonzack]

Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.


Disclaimer before clicking the below link: The youtube video contains some foul language. I copied at a time stamp past that portion of the TED talk.

Been following this thread, with a TED talk (http://www.youtube.com/watch?v=_x1jJvp1ERs&feature=player_detailpage#t=1222s) by Anthony Robbins ringing in my head. Everyone reacts to all situations differently. Some people will find silver linings to anything. Some people will point fingers. Some want heads to roll, others want to give a hug.

We are role models. Not just our mentors, our entire community to the rest of the world. However you react, make sure whoever is watching you is inspired by it in a positive way.

[Brandon_L]

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

[Steven Donow]

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

I'm obviously not affiliated with them, but I believe at regionals(CT and Alamo specifically) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

[MikeE]

I'm obviously not affiliated with them, but I believe at regionals(CTand the other one that they went to that wasn't Lone Star... Alamo?) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

The CT eliminations (in week 5) saw multiple robot loss of control events from several good teams, some of which were clearly robot system issues (IIRC faulty battery connector, USB hub disconnection) but also others which had no clear cause identified at the time.

As Steven suggested it's possible that 118 suffered from the same subtle and unfortunate system interaction bug that was discovered before the Einstein investigation weekend.

But one of the effects of the Einstein revelations on us all as a community is that now there has to be some suspicion that a similar act of interference occurred at Hartford.
Innocence lost...

[Greg McKaskle]

I did not attend CT, but worked with 118 the following week in Houston, trying to identify the issue. I looked at the logs from CT and they looked very similar to Einstein. It is very likely that the sensor connection led to the failure in Houston and the ones in CT as well. They made code changes in Houston, I don't believe they did so before Einstein, and I don't believe they introduced the problem between divisions and Einstein. The code issue was present, lurking for a long time.

If the sensor connection had never failed, the loop in the init code would do what it normally did and the robot would have operated wonderfully.

If the sensor connection had failed permanently, they would have hooked up a complete debugger in the pits, located the loop and the sensor, fixed them both lickity-split, and operated wonderfully afterwards.

But the sensor connection apparently failed just a few times during the season. Perhaps it was brought on by the cart or the loading or reset procedure, or vibration, but since it didn't stay in a failure state, the chance to debug was fleeting. Additionally, the sensor wasn't used and the init code wasn't executed unless the testing included the auto-tele transition. I believe this was another factor that influenced how team 118 interpreted the cause. Bugs that are difficult to reproduce are incredibly frustrating in all disciplines.

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well. Most things in the world don't work the first time, they interact in ways you didn't predict, and they may change over time or under different conditions. Managing that chaos is a part of what engineers do.

118 is a great team that builds great robots, but this time both Murphy and Achilles had their influence. I look forward to working with them in the future.

Greg McKaskle

[JamesTerm]

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well.

Yes I agree... we have a completely different simulation to run our code. To add to this... I never... never write an infinite loop (and hope for some breaking condition). There should never be a case for that, but I see programmers do it (even some of the best).

Just to be clear... I define an infinite loop as this:
while (true)
{
if (I hope this works) break;
};

You can always do this instead

while (timeout++ < threshold)
{
if (I hope this works) break;
}
if (timeout >= threshold)
{
assert(false);
error recovery here;
}

Also any thing inside the autonomous loop (in this case) could check for
(IsAutonomous() && !IsDisabled())
to determine if autonomous is still happening... in my previous entry it is absolutely critical to exit autonomous loop ASAP... otherwise you'll lose telop connection for a good chunk of the match. (This is worth repeating).

If anyone knows of that issue being fixed please let me know... According to this report it sounds like it still is the same as it was last year.

[Arefin Bari]

After reading through the Einstein report and this thread several times; I had to digest all the information before making a post.

It is very unfortunate what happened on Einstein. I was home watching as my wife was screaming at me because we were supposed to get to dinner. I was very excited because most of the team that were on Einstein this year are my “friends” and they have done enough for team 108 for the last several years. I was ready to watch them tear it up on the field instead I sat there staring at dead robots.

I have nothing more to say than I am disgusted by the individual’s action. The teams affected have every right to be upset and frustrated. Heck, I still hold grudges against a mentor that came into our pit couple years ago and thought it would be gracious to disconnect all the pwm cables from the jaguars. It is unfortunate that situations like this happen in our community. I bet the Einstein teams attending IRI this year will bring their A game and I can’t wait to see them all compete in person.

[Justin Ridley]

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

We did not change any code before Einstein. As Greg states, this was likely an issue that was with us all season. While we know the gyro reporting bad data was the start of our error chain, we’re not 100% on why we got that bad data. The way we could replicate this failure post Einstein was by unplugging the gyro (no data = bad data). The report states a faulty crimp is the most likely cause. This may in fact be true, however, it’s a little hard to accept due to seeing the failure so infrequently and having no issues in between. For instance, we saw one of these failures on practice day of Houston. Without touching the robot, we were able to run another practice match immediately after with the exact same robots on the field. This match ran without issues.

After this practice match in Houston, we replaced the cRIO. We didn’t see another occurrence of the problem until the first match on Einstein. Obviously we thought the cRIO fixed the problem and there was no need for further troubleshooting.

Looking back we can only assume that the problem outlined in the report was the cause of the one failure we had practice day in Houston and the two semi-final matches in Connecticut. The second leg of the error chain was part of the code. After Connecticut, our programmers literally spent hours looking for a spin loop without an exit. It simply wasn’t obvious and they didn’t find it.

As Greg alluded to, one big lesson learned for us is how to better perform our full robot checkouts, which we do before and after every match. We did NOT include a run through autonomous mode during these checkouts, which meant we would never have seen this failure. Had we done this, we may have seen the failure more often and been able to better diagnose it.

We learned some valuable lessons from this, and are very thankful to Greg and the FIRST team up in NH for their work with us and all the Einstein teams.

We know how horrible it feels to have our robot not compete due to reasons that ultimately were our fault. We’re sure it must feel much worse to those teams who were brought down through no fault of their own. We can’t begin to express how saddened we are to know that this happened to our fellow competitors.

[Al Skierkiewicz]

Justin,
It is my understanding that your code has a method for checking all sensors as part of the initiation sequence. When the gyro reported bad, the code stopped. We tested this in NH and received the same result when the gyro wire was pulled. During Einstein, we replaced your Crio, DSC, DSC cable and everything else I could think of without effect. We checked for power issues, bent cables, as much as we could in the amount of time we had. I have to tell everyone that 1114 and others were offering hardware and assistance during this time as you would expect. (remember Thunder Chickens from a few years ago?) The FTA and CSA came over to lend support as well.

[Al Skierkiewicz]

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

[Sam124]

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

I couldn't agree more with every point that you make, Al.

[KelliV]

Hi guys, I've waited awhile to post so that I could clear my head and here goes.

Al said pretty much everything I would want to say about the intentional interference so I won't even address that.

I do, however, have a suggestion for those of you who are significantly smarter when it comes to technical mumbo jumbo that I am. Several of my friends have looked at the report and come away with the same reaction as I have, that being... huh???

To help others understand what really went wrong can someone sum up, in easier terms, what happened technically with the robots. Imagine what you would say to someone who isn't in FIRST when explaining what's wrong. I think this is part of the problem as to why many are seeing, as Dr. Joe put it, the tree rather than the forest.

Everyone out there understands what happens when someone interferes, but not many people other than FIRSTers really understand CRio problems or CPU usage. So on that long drive to IRI if someone could make sense of a lot of this it would really help a lot of people out.

[simpsonboy77]

KelliV,

I'll try to simplify it as much as possible.

The first tests (1-6) were to test FMS. There could be many issues with the hardware used to run FMS, so this was to test the software FMS uses, hardware, and peripherals. I don't know exactly what they were looking for, but I'm guessing just stability issues. A faulty router could cause drop outs etc. Basically you are looking for something, you don't know exactly what it is, but when you find it you will know. This is the approach for intermittent issues, poke the system until something gives.

The client authentication issue works due to issues in the D-Link DAP-1522 Revision A hardware. This is the wireless bridge that you stick on your robot. This can only manifest itself when paired with the newest firmware of the access point used during competition. No team would have control over this as it is FMS controlled. It is a Cisco 1252. The firmware was changed in week 4 because of an issue where the 1252 would reboot while setting up a new match. I am going to guess that this was done to decrease setup time between matches. However Cisco did not verify the setup in entirety. Had they done this, the issue would have been caught, and likely fixed in firmware.

As for which device connects, it honestly does not matter. What matters is how often the device connects to the network. It is relatively easy to write a program that will run on a laptop to imitate this. What is worse is such a program can be written and run without user intervention, so the person next to you in the stands would not know if you are attacking the field. I will skip what happened at Einstein as that is not as technical, and I am somewhat pressed for time making this post.

De-auth attacks are very effective at causing denial of service. Basically an attacker can spoof the access point, and tell the robot's bridge to reauthenticate and associate. This takes time, and it is trivial to send this packet upwards of 400 times a second at one robot. I may have misinterpreted this, but an attacker can send 400 packets a second at a single robot for 89 seconds and not be detected. Additionally an attacker can trickle deauth packets for the entire match and not be detected. A trickle of 3-7 packets per second will cause some network congestion for that one robot. This is not so much a vulnerability in any hardware or FMS, but more so a vulnerability in 802.11. There are other vulnerabilities such as TKIP session hijacking in TKIP (we use AES), and hole 196 in WPA2 AES. It becomes very difficult to fix these bugs as they are beyond the scope of FMS and FIRST.

The robot specific issues are to show that while it may have been an attack the entire time, these other issues likely contributed to failures.

A high CPU usage could hinder the cRio's ability to send packets to communicate with the driver station. If the cRio is trying to process images, that is time it is not using to communicate with the driver station. If enough data gets buffered and not handled, this will cause drop outs. Think of it as someone reading off several phone numbers and you are writing them down. If the person reads them too fast, you cannot remember all of them. The ones you forget are analogous to dropped packets.

Further down in the report they mention QoS, or quality of service. This will let FMS limit how much bandwidth a robot can use on its own. There is ideally only 300 mbps of bandwidth available amongst 6 robots. Two uncompressed video feeds can eat up much of this. This will prevent robots from starving other robots of bandwidth. It can also be configured to prioritize driver station packets over custom and video packets. This will likely be done off of ports, however I do not know for certain.

As for additionally monitoring, computers can be setup with programs such as wireshark to listen to ALL communication. This will be useful to spot attacks that aren't listed and provides comprehensive logging. Wireless cards can sometimes be put into a mode call promiscuous mode which allows them to listen to data even not directed at them.


I hope that helps, I tried to clarify as much as possible. The report was extremely thorough. I read it first and thought it was missing some important data points, however on a second and third read through, I realized I just missed them >.<

-Garrett

[Greg McKaskle]

Garrett's email explained things pretty well I thought. If there are more specific questions, this is a pretty good place to ask.

Regarding CPU usage, the code was inspected to ensure the reason was well understood. 100% usage itself will not cause communication issues, but it may be a symptom of something else. The DS logs of each robot for each Einstein match was also inspected for CPU, battery, communication quality, timing of dropouts, and other odd patterns in how control packets were processed.

Greg McKaskle

[BigJ]

snip

Good explanations of concepts, but a quick glossary:

Spoof: When a client or user "spoofs" another, it provides the communication destination with information that leads the destination to believe that the client or user is one that they are not.

Packet: Information gets sent over the network in packets. Slow or missing (also called dropped) packets make the driver<->robot appear slow (or laggy) or even disconnect.

[EricVanWyk]

The super simplified version:

Robot: Hey FMS, want to chat? Here is my key.

FMS: Looks good! Lets chat!

"Individual": Hey FMS, want to chat? Here is my key.

FMS: Hey, that key is wrong. I don't want to talk to you.

Robot: Aww, the FMS doesn't want to chat with me anymore. Was it something I said?

[Anupam Goli]

The super simplified version:

Robot: Hey FMS, want to chat? Here is my key.

FMS: Looks good! Lets chat!

"Individual": Hey FMS, want to chat? Here is my key.

FMS: Hey, that key is wrong. I don't want to talk to you.

Robot: Aww, the FMS doesn't want to chat with me anymore. Was it something I said?

In the seriousness of the discussions going on, this one seriously made me laugh, and is a great analogy.

[JaneYoung]

These last few posts that took the time to answer Kelli's request - are proving very helpful. Thank you so much.

Jane

[EricVanWyk]

Super Simple Deauth:

"Individual": I am the FMS. I don't want to talk to you.

Robot: FMS, is this true? I still have stuff for you.

Actual FMS: No, what gave you that idea? Are you feeling ok?

Repeat.

Field Monitor: FTA! We need a counselor!


Result: Communication slows down, and can sometimes drop out entirely. This is a possible attack that was largely ruled out. If someone were to launch this attack, the Airtight system would count the number of "go away" messages and display an ugly warning when it hit a certain threshold. We found that that threshold was too loose, so we're tightening the chain. We can't set the threshold to 1, or we'd get false positives several times a match - it is a valid message to send.



Super Simple Priority Inversion:

Robot: Gyro, please reset yourself and verify proper operation.

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...

Field: Robot, I keep telling you to reboot, but there there is nowhere for me to put my message to you.

Result: The robot code locks up. The cRIO's safety mechanisms kick in and prevent the bot from moving. This is actually a good thing. The bad part was that it wasn't able to get the command to reboot and try again.


Super Simple Network Tables Flood:

Dash Board: Hey Robot, here is some new data.

Dash Board: Robot, did you get that data yet? Hello?

Robot: Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks!...

Other Robots: Will you please shut up?

Result: A few seconds of extra lag. All of those unnecessary acknowledgements eat up radio time, and can cause brief control losses. The plan is to put fairness guarantees in place that prevent this type of error from affecting the other bots.

[Racer26]

@EricVanWyk: AWESOME analogies. Almost totally accurate too.

[Akash Rastogi]

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...


This might be the funniest thing I've ever read on CD. Thank you.

[Greg McKaskle]

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

[BigJ]

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

Who watches the watchdogs?

[s1900ahon]

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

I'd rather EJvW draw some (xkcd-like) stick figure drawings in the page corner so we can just flip the pages really quick to animate the sequences.

[ratdude747]

I'd rather EJvW draw some (xkcd-like) stick figure drawings in the page corner so we can just flip the pages really quick to animate the sequences.

THAT would be so epic... :D

[techhelpbb]

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

I suppose one should ask what they should expect for a response if they do this?

[EricVanWyk]

Do we have any volunteers for illustrator?

The weird part? This is how I actually think about the control system. Engineering is so much easier when you give all the components personalities. Then you just watch the drama unfold and try to fix it.

[Tetraman]

Super Simple Priority Inversion:

Robot: Gyro, please reset yourself and verify proper operation.

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...

Field: Robot, I keep telling you to reboot, but there there is nowhere for me to put my message to you.

Result: The robot code locks up. The cRIO's safety mechanisms kick in and prevent the bot from moving. This is actually a good thing. The bad part was that it wasn't able to get the command to reboot and try again.


Thank you very much for the laugh.

I made this for you:

http://i49.tinypic.com/ac9s5.png

[Ether]

I made this for you:

http://i49.tinypic.com/ac9s5.png

Cute.

[Astrokid248]

Thank you very much for the laugh.

I made this for you:

http://i49.tinypic.com/ac9s5.png

This is perhaps the greatest thing I've ever seen on CD.

[Chris Fultz]

I suppose one should ask what they should expect for a response if they do this?

From the letter from Jon Dudas -


Once again, we want your feedback. FIRST is a community that gets stronger as we work together to solve problems. We are particularly interested in your thoughts on the “Next Steps” section of the report. Your comments and ideas can be sent to 2012frcfeedback@usfirst.org . While we may not be able to respond directly to every e-mail, I guarantee that each will be read and fully considered.

[IKE]

Do we have any volunteers for illustrator?

The weird part? This is how I actually think about the control system. Engineering is so much easier when you give all the components personalities. Then you just watch the drama unfold and try to fix it.

So, would you consider yourself a group counselor?

[dyanoshak]

I'd rather EJvW draw some (xkcd-like) stick figure drawings in the page corner so we can just flip the pages really quick to animate the sequences.

XKCD already has a character that could play the "Individual"... The Man with the Black Hat.

https://lh5.googleusercontent.com/-tAO45UPoIwQ/UA2bcOLaUSI/AAAAAAAAAPQ/9LodXOoYKYI/s256/xkcd-maninblackhat.png

Note: This is a cropped frame from a XKCD Comic #929 "Speculation" (http://xkcd.com/929/).

[dyanoshak]

So, would you consider yourself a group counselor?

Actually, I hear that EVW was an LCPC (Licensed Clinical Professional Counselor) for about 20 minutes last year...

[plnyyanks]

XKCD already has a character that could play the "Individual"... The Man with the Black Hat.


Black (http://xkcd.com/81/) Hat (http://xkcd.com/190/) Guy (http://xkcd.com/954/) is (http://xkcd.com/72/) perfect (http://xkcd.com/325/) for (http://xkcd.com/925/) that (http://xkcd.com/972/).

...okay, maybe I have a xkcd addiction

[gracie.]

Black (http://xkcd.com/81/) Hat (http://xkcd.com/190/) Guy (http://xkcd.com/954/) is (http://xkcd.com/72/) perfect (http://xkcd.com/325/) for (http://xkcd.com/925/) that (http://xkcd.com/972/).

...okay, maybe I have a xkcd addiction

I love this one.... (http://xkcd.com/689/)
I think it'd be great to have some simple break-down of the report for people. I read it, though a large chunk of it was way over my head- this whole dialogue thing is a great way of explaining it :)

[Anupam Goli]

Thank you very much for the laugh.

I made this for you:

http://i49.tinypic.com/ac9s5.png

Someone spotlight this post!!!

You should totally start a webcomic detailing FIRST events in a humorous, yet understanding way.