[FRC Blog] Einstein Report Released
 

http://frcdirector.blogspot.com/2012...-released.html:

Re: [FRC Blog] Einstein Report Released
 

I would just like to say that FIRST handled this entire situation in exactly the perfect way.

Re: [FRC Blog] Einstein Report Released
 

Wow some shocking and disappointing things in this report.

Re: [FRC Blog] Einstein Report Released
 

Skimmed through, need to thoroughly reread.

As of now, impressed with the thoroughness (as well as their decision to give automatic champs births along with waiving the entry fees for the first event).

Unsure how I feel about some of the descriptions of inappropriate actions.

Re: [FRC Blog] Einstein Report Released
 

I can't believe what I am reading, who would do that to kids.

Re: [FRC Blog] Einstein Report Released
 

I'm quite shocked by this, why would anyone do that?

Really impressed with the way first is handling this though, great move making sure the teams from Einstein get to come back to championship in 2013 and also waiving their first play fee.

Re: [FRC Blog] Einstein Report Released
 

Does the report specifically say what type of interference took place? I see references to potential RF interference in a few of the paragraphs about 25, but that's about it.

Also, does anyone know if the guilty party stepped forward willingly, accidentally interfered, or anything else?

I can understand why they would want to be vague about who did it, but at the same time I want to know more.

(See Alexa's post below)

In the end, though, I'm impressed with how well they handled the situation.

Re: [FRC Blog] Einstein Report Released
 

After reading through this, I believe FIRST should either replay the matches or declare no champions this year and allow all 12 teams free payment for next year's championship.

Re: [FRC Blog] Einstein Report Released
 

And the usage of a Galaxy Nexus will only fuel iPhone fanboyism:o

But in all seriousness, pretty disappointing that someone would do that.

Some slight skimming through shows that this is an interesting read.

But reading the planned fixes/changes has me excited...new radio...more documentation on DS components AND field components...looks like next year will be interesting to watch things unfold as FIRST implements new changes/monitors everything (hopefully) more closely...

Re: [FRC Blog] Einstein Report Released
 

I suggest reading the Failed Client Authentication on Einstein section. It explains everything regarding that. ;)

Re: [FRC Blog] Einstein Report Released
 

Just finished reading through.

Wow. I'm impressed by how detailed the entire report is, and at how well FIRST has handled the whole situation.

Re: [FRC Blog] Einstein Report Released
 

Hmmm. Intentional interference... sounds like FRC is really starting to look like a "varsity" sport.

Aside from Einstein having issues there were tons of radio and cRIO issues at regionals. It really is time for FIRST to look at a new system.

Re: [FRC Blog] Einstein Report Released
 

Thank you to the teams and persons and FIRST for the exhaustive effort.

Re: [FRC Blog] Einstein Report Released
 

I'm happy to see that the electronic power supply issues were considered and annotated. All the testing I did at off season events showed few and far between issues related to the AP/router power supply with some issues with the AP/router power supply (from the battery to the radio power input) taking a bit longer than might be a good idea to reach full output regulation from cold power up (how sensitive the AP/Router is to this is variable and some units might be effected). The time the AP/router supply takes to reach full regulation voltage may leave room for the routers to come up in strange states. This makes sense, all the teams suggesting they fixed their problems when they powered down the AP/router a second time and then back on again after powering up the robot when they initially got on the field (how often it happens though would be quite hard to determine).

The person attempting to connect to the field network is bad news and as long as critical field functions are connected to a public common network FIRST will run this risk. Lucky for everyone this was done line of sight to the effect. Frankly the attack (lack of common sense/failure to communicate intentions) in question could easily have been done any number of clandestine ways that would not have been noticed (I am not going to list them out of concern that someone will try them). Unfortunately it's really easy for someone to create something that will attempt to connect to the field network while aggressively hunting for Internet connections.

I see that they considered the antennas for the field and a few placements of the field equipment. I would have thought they would have tested that further with the robot side equipment power measurements as both sides transmit and receive but apparently that did not happen. Suggestions on how to best optimize the range of the KOP standard robot mounted AP I should hope will find their way into next season.

Overall, I'm satisfied that they've done all they can with what they have as far as a test is concerned. I'm not sure I'm convinced that this one person was the cause of so many headaches however. Einstein was hardly the first time connectivity issues surfaced that were not readily explainable by power supplies issues or programming.

My thanks to FIRST and all those that have worked so hard on trying to make sure this does not happen again.

Re: [FRC Blog] Einstein Report Released
 

I believe Bill mentioned that the NI contract will be up for the 2015 season.

- Oliver

Re: [FRC Blog] Einstein Report Released
 

The organization did what they were expected to by members of the FRC community and more, even with the news arising that intentional interference took place. my eyes really widened when I read that. I'll finish it after I let that thought sink in.

It would have been a poor option to take away the award from the kids and their organizations that support them, but it's great that they extended the berths to all teams on Einstein and paid for the teams to register in 2013.

Re: [FRC Blog] Einstein Report Released
 

ಠ_ಠ So, asides from the intentional foul play, it seems as if their was a lot of errors on the teams themselves. Seems like 987's problem was that there was a deadlock with their software. So, that's a shame.

Re: [FRC Blog] Einstein Report Released
 

I think it is important to remind readers to not be too shocked and awed by the fact that interference happened; I think it should be viewed as another thing that is potentially a problem with the control system.

My point being; don't let this news be the face of the issues with the control system. The fact that these elite teams were able to have such problems with the system leads me to say that perhaps this is not the ideal system for FRC.

This report is great and revealed many problems teams may have dealt with all season.

+0.02

Re: [FRC Blog] Einstein Report Released
 

I don't know if you are joking or not, but the ramifications of this would be ugly, at best. The current step was the correct one.

Meh, teams have thousands of dollars of control system laying around. If we must, then I suppose we'll go to a new system, but only if we really have to.

Time to read the detailed report,
Sunny G.

Re: [FRC Blog] Einstein Report Released
 

WOW thanks you to FIRST for releasing this out to everybody I appauled that it hopefully wont happen again.We had major issues with the FMS @ Cheaspeake regional and saw they were used on Einsten field to.Whish FIRST would have done something to the fields that the electronics came from to instead of just from the Einsten matches

Re: [FRC Blog] Einstein Report Released
 

Wow. Intentional interference. Shame they decided to let the results stand after that. I'm sure 1114/2056/4334 are left with a rather sour taste in their mouths.

I read the whole report, and while a number of possible issues were identified for the various teams, only 118's dropouts were really confirmed to be caused by team equipment/configuration.

I will say that for the FCA attacks on 1114/2056/4334 to be intentional, someone would have had to have figured out that the FCA vulnerability existed, and based on the way the report talks of it, it seems that this only existed from Wk4->Championship, and only on robots with rev A hardware. Curious, indeed.

I still stand by my original assessment from April that the proper way for FIRST to have handled Einstein is to award all 12 teams 2012 FRC Champions, in lieu of being able to play a fair set of matches.

Re: [FRC Blog] Einstein Report Released
 

I am just as shocked about the intentional interference as everyone else is, but I really would like to extend sincere thanks to the twelve teams, the volunteers, the experts, and most importantly to FIRST and FRC for taking the time to deal with this in such a professional and thorough way.

Re: [FRC Blog] Einstein Report Released
 

David,
If you read the full report carefully and pay close attention to the following:

987 – While testing of this robot revealed programming issues that could cause higher than normal trip
times, or the cRIO controller user code to lock up, none of these issues were found to cause a command response failure. Which was the cause of our "dead" time on the field). The duration of these losses were too short to be attributed to a cRIO or robot radio
reboot, but fit well with the symptoms of failed client authentication.

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

Re: [FRC Blog] Einstein Report Released
 

Yes, I admit, I have not read carefully enough. Still, that's a shame IMHO. A deadlock anywhere is not good.

Re: [FRC Blog] Einstein Report Released
 

Barry Bonds used steroids, however in the record books he is still listed as hitting the most amount of homeruns.

Just thought I'd throw that in for comparison. Also, it isn't known(well, to the general public, i'd assume those involved do know) whether or not the interferer was directly targeting a specific team; just that he was only SEEN targeting 2056 directly.

IMO intentional interference being reasoning to change the results is an invalid point. As stated, FIRST is now more like many varsity sports :D

Also of interest was that it seems like 118's issue was also code and that no team on Einstein was using Labview. For reference, do 1717 or 469(had comm issues in their semis) use labview?

Re: [FRC Blog] Einstein Report Released
 

Worth noting: I expect the FRC Community at large to be wholly unsatisfied with HQ's decision not to replay the matches (and/or declare all 12 teams Champions in lieu), given the circumstance of intentional foul play.

Re: [FRC Blog] Einstein Report Released
 

1717 Did not use labview for sure.

I don't think you can blame FIRST for not replaying the matches... Replaying Einstein 2012 would be asking for trouble. Teams or the community shouldn't be unsatisfied. The alliance of 25/180/16 were darn solid. 25/180 hit their shots and could triple. 16 was by far too freakin amazing at stealing/feeding. And the alliance had a really solid auton/endgame.

Teams and the community should be stoked that FIRST took a giant step forward and are working towards figuring out issues with the control system. They even published a very nice paper. Kudos to FIRST.

-RC

Re: [FRC Blog] Einstein Report Released
 

I have a feeling that while there was intentional FCA going on, there's a large potential for unintentional FCA as well, which they didn't mention whether or not had been tested. Many teams configure their practice network at the shop with the same SSIDs that the field uses, and if they used a different security key, any phones/tablets/laptops that happened to be active during the games and had been connected to the network at home could have been attempting connections in the background. I actually caught our team's programming laptop trying this once during week 2.

Also, I was surprised to notice this year that SSID broadcast was enabled on the field APs, making the team networks show up on every laptop in range. I don't remember it being that way last year.

On a somewhat related note, what would people think about having a CSA check over code as part of inspection to counter the kind of user code issues that were seen in the report. It wouldn't have to be a pass/fail kind of thing, but having a more well-trained pair of eyes to provide suggestions to the team and something for field staff to go on when a robot misbehaves would probably go a long way

Anyways, excellent report by FIRST, and I think bandwidth caps are a great idea. I hope they don't take the partial involvement of the BeagleBone as a reason to ban non-KOP electronics though. They really open up new possibilities for teams.

EDIT:
I don't know about those two, but after those semis 68 had unexplained comms issues in the finals, and they do use labview

Re: [FRC Blog] Einstein Report Released
 

Impressed by FIRST's response, my faith in FIRST has been restored. So ridiculous what the hacker did though. I'm curious as to what his relations are to FIRST/any teams..

Re: [FRC Blog] Einstein Report Released
 

While I don't think there's a strong case for replay or any change to the officially announced results of the 2012 FRC Championship, I do think it's worth noting that prior to their being knocked out, the Archimedes alliance was the only one suffering problems likely caused by FCA. If the interferer had an agenda, it seems that 1114, 2056, and 4334 was the primary target.

Re: [FRC Blog] Einstein Report Released
 

I am definitely happy to see first taking steps to resolve the issues they discovered, especially with better documentation for advanced coding, the investigation of a new radio, and a fix for the NetworkTables issue.

Re: [FRC Blog] Einstein Report Released
 

Wait, isn't this the reason why you debug and profile your code?

Re: [FRC Blog] Einstein Report Released
 

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

Re: [FRC Blog] Einstein Report Released
 

Agreed, along with the limited time and constant updates.

-RC

Re: [FRC Blog] Einstein Report Released
 

You will notice in the report that it also seems like 987 suffered the same problem in Final 1 and 2...

Re: [FRC Blog] Einstein Report Released
 

I cant help but believe that the Curie Curse had something to do with the Curie alliance in the finals.... :ahh:

Re: [FRC Blog] Einstein Report Released
 

Well, I would disagree. From the language of the descriptions, it seems as if the problems would arise from general use. Sure, you'll definitely have to try hard to get certain conditions, but debugging and profiling your usual usage conditions is enough IMHO. Like the 100% CPU usage should have easily been caught.

edit: perhaps, it's just me. I like spending time with my robot.

Re: [FRC Blog] Einstein Report Released
 

If I were the boss, I wouldn't have released this report on Friday the 13th.

Re: [FRC Blog] Einstein Report Released
 

True, but 987 didn't have those problems in the semifinals, suggesting that the Archimedes alliance was the higher-priority target. Of course, you're right, the fact that 987 suffered the same problem means that you were probably targeted as well. While it's regrettable that this happened to anyone, but it seems fairly clear that there was a pattern to the chaos which began with 1114, 2056, and 4334's issues.

Re: [FRC Blog] Einstein Report Released
 

I think FIRST has handled the situation well so far. FIRST could have easily turned a blind eye and tried to sweep the problem under the rug. What would replaying the 2012 championship matches really solve? Nothing. In my opinion, giving the 12 teams automatic invites to the 2013 Championship as well as waiving their first play registration is enough consolation.

The issue isn't just trying to connect to the network, the issue is trying to connect to the network - while entering an incorrect WPA key:

Lets hope that FIRST continues to learn from Einstein 2012 and makes changes to the control system to prevent this from happening at future events.

Re: [FRC Blog] Einstein Report Released
 

There were several things I got out of this paper, especially as an engineer working on engine controllers:

-The Smart Dashboard had a bug which was exploited which caused a deadlock. While all software has bugs, it should also be tolerant of failure, meaning the rest of the system should have been designed to operate (possibly in limited quantity).

-The Smart Dashboard was mentioned numerous times relating to increased network load, especially the funny 1-byte packets.

-The VxWorks operating system handling of the packet buffer seems exceedingly poor. Many other forms of communication (e.g. some CAN stacks) dump old packets with the same ID when they are added to the buffer, this seems like the right move (at least on UDP).

-The boot time of the cRio was mentioned to be 24s minimum.
--I am currently working with an engine controller that can reboot the application software fast enough to not stall the (Diesel) engine while it is running.

-The nature of 802.11 makes it a poor choice for this kind of wireless communication.

I will not comment on anything else.

Re: [FRC Blog] Einstein Report Released
 

David, the deadlock is within the WPILibrary and we have been helping them resolve it. In fact, just today we were at the shop with our programmer Brandon as he finished up test programs for FIRST to use to reproduce two bugs we encountered in the smartdashboard support code this season.

These bugs did not affect our robot on Einstein or in any match all year because we avoided them. Working towards getting them fixed for next year is just a side benefit of the New Hampshire meeting.

Re: [FRC Blog] Einstein Report Released
 

If you read the document carefully, it states:
Later on, it states that the likely cause of 1114's issues was FCA.

Also:
The 100% CPU usage was not found to be the cause of their issues so debugging that issue probably would not have helped (much).

I really don't think many of the issues described in the document can be attributed to teams not debugging carefully enough.

Re: [FRC Blog] Einstein Report Released
 

You mean like they have been doing since they introduced this system? This report is several years too late, and even now it fails to address many issues. It is a shame that this report does nothing to address the issues at have been occurring at regionals since the introduction of this system.

Re: [FRC Blog] Einstein Report Released
 

Issues in years prior may have had nothing to do with the deauth attack- only one radio and one AP that they tested were vulnerable. Older radios may not be vulnerable. I don't know if the AP has been changed.

Re: [FRC Blog] Einstein Report Released
 

And that is precisely why this report fails at its root. I care very little about the problems that affected 12 teams when there are other problems with the control system that affect 100s of other teams.

Re: [FRC Blog] Einstein Report Released
 

This made me laugh in the midst of being upset about the news. Thanks for that haha.

Re: [FRC Blog] Einstein Report Released
 

I think this thought process is very dangerous, before you know it you are pointing the finger at certain teams that were on Einstein. It could have been so many iterations of people and pointing the finger at someone specifically the wrong someone is not what FIRST is about.

Re: [FRC Blog] Einstein Report Released
 

It says the exact opposite of that.

Re: [FRC Blog] Einstein Report Released
 

Yeah, my bad. I fixed my post. :)

Re: [FRC Blog] Einstein Report Released
 

FIRST Hall of Fame Team 1114, Simbotics, would like to thank FIRST for thoroughly investigating, addressing and documenting the robot failures that took place on the Einstein Field at the 2012 FIRST World Championship. We would especially like to thank Frank Merrick and his staff for constantly communicating with us through the process.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

Re: [FRC Blog] Einstein Report Released
 

When I saw the email blast come through on my phone, I seriously sat in my car for 20 minutes or so reading the PDF. My thoughts:

1) I applaud the team for being this thorough in their methods and sharing all the steps they took (and for ruling out things we would normally hold up as the cause).
2) I believe FIRST has done right by the teams involved. Nothing is gained by replaying the matches.
3) I thank the teams for checking their own systems and code to discover issues of their own.
4) Shame on the individual who attempted this stunt. It hurts that FIRST has to have its answer to baseball's permanently ineligible list. We should all expect a higher standard.
5) I hope some element of this process is framed and put somewhere fairly prominent in FIRST Place. I think this entire story contains elements of the FIRST experience both at its worst (see point 4) and at its best (see point 3). Much can be learned from both.

Re: [FRC Blog] Einstein Report Released
 

That was one well written report, and certainly insightful to the whole FMS process. It gave our team a thing or two to look out for in the up coming seasons, as well as possible things to look immediately at for debugging.

As far as the repeated notions of a "tainted" win and questions of replay or total recall, I believe that is out of the question. 16, 25, and 180 won the 2012 FIRST Robotics Competition World Championship, and should not be forced to defend their title.

Re: [FRC Blog] Einstein Report Released
 

Even though I think that the teams that had this happen to them deserve to know who did it and why, I foresee what you are asking turing into a similiar mindsight as the Jerry Sandusky case. That is, that the perpetrator did it full knowing what he was doing and what the consequences could be and forgoing the effect it would have on others, but it in turn not only effected him but also Penn State; the point im getting at is, putting forth the person and team publicly would do far greater harm to the team and their image amongst their peers and the FIRST community. I think a much better solution to what Karthik wants, and is justified in wanting, is to have all the parties involved meet so that everyone can be satisfied as to reasons why, punishment, etc... But I do beleive that this shouldnt be known to the masses.

Re: [FRC Blog] Einstein Report Released
 

The larger issue than who did this is how was the system allowed to get to the point that it was possible at all. Let's worry about all the other D.O.S. (denial of service) attacks we have yet to find.

Clearly changes need to be made. It took extraordinary effort on the part of too many people to resolve even these issues to this point.

It still goes back to the assumption that the system is above flaw and that assumption being incorrect. In this case the system has a security issue and an active exploiter. Take if from me: you can look for and fix security issues before they get exploited as best you can or you can wait until they cost you reputation, resources and opportunity. Had they even profiled the issue before hand they could have dramatically reduced the chaos after the fact (if you don't fix it at least acknowledge it exists at the remediation level).

FIRST needs to consider a secondary channel in their control system if they can't more fully profile something like the WiFi system they have now. It's the logical alternative to pumping all the data into a single publicly exposed communication system. FIRST did have the AirTight equipment on site but clearly that alone wasn't sufficient to keep a lid on this issue. Additionally the FIRST report oddly doesn't discuss that the AirTight hardware did not produce for them sufficient warning of this issue or whether they consider that something they need to pursue (it was the assumption that this additional monitoring was sufficient to keep intentional interference from happening at all).

I give FIRST credit for the heavy work they put into this. I'm just not convinced that this won't happen again if someone intends to deny service to the field. Nothing I've seen in the recommendations will stop it. I can think of 3 ways right now that if I wanted to render all the field robots dysfunctional I could and it would be nearly impossible for them to discover it. Even if the sudden failure was recognized as intentional disruption the cost in resources to weather such interference is unacceptably high. A better solution needs to be found for this.

Re: [FRC Blog] Einstein Report Released
 

The PSU case is different-many university officials actively participated in a cover up. I don't think any comparisons should be drawn between the two in any case, given the difference in severity.

Re: [FRC Blog] Einstein Report Released
 

I know the report is long, but I urge everyone who wants to make a statement about the report read the entire report first. There have already been multiple posts in this thread containing false assumptions.

Re: [FRC Blog] Einstein Report Released
 

If one of the winning teams had to do with this interference, or even any of the teams who played in Einstien, I would have agreed (of cource, not giving the interfering team any title). Since Jon Dudas's letter clearly states none of these teams were involved, I think FIRST made the right decision.

Re: [FRC Blog] Einstein Report Released
 

I saw this already as I've read it three times. I consider my statement correct.

They assumed that the AirTight system was going to trap such attempts and they assumed wrongly.

The point remains that it's highly likely that there are many other things that AirTight won't trap (2 other that I am acutely aware of) that are not already discussed in this report.

In short, yes you can read this that I limit my concerns to deauth but frankly the solution isn't to fix one problem in AirTight and think that it's now fool proof security. That's the sort of thinking that created the problem.

They need to completely reconsider how they transit the really important traffic. Custom solutions in their context could mean anything (including loading existing hardware with DD-WRT or OpenWRT which itself is not free of exploit).

The long term risk is that all the focus sits on this particular vector of attack and polite offloading of all security concerns to AirTight continues to leave exposed other vectors of attack. It's not about blame at this point. FIRST has gotten more blame than they deserve in some ways and AirTight doesn't make robot WiFi security products specifically for FIRST. It is just about suggesting that AirTight has only that issue which is wrong and is what this report basically does. It's not the be-all-and-end-all of security solutions (almost nothing ever is regardless of what sales says).

Re: [FRC Blog] Einstein Report Released
 

Everyone,
Now that the report is out, I urge you to read it in it's entirety, twice. Then sleep on it. Please do not cherry pick from specific parts of the report and draw conclusions about the system, the robot or the hardware in general. Note that each team is handled separately so that you can understand specific issues that occurred during the matches on Einstein. Please do not generalize a statement from a specific team report to indicate this occurs for all robots. (e.g. The Crio reboot time for Team 233 only was found to be 24 seconds.) Also, I urge everyone to stop using the generic term "communication failure" to describe the Einstein issues. The report is detailed enough that using that term is not descriptive of what actually occurred.
I would like to personally thank everyone involved in the Einstein weekend investigation. Each person was committed to finding answers, sharing data, and coming to a conclusion that would be of help to all of us. This will be one of those high points I refer to when asked why I continue to participate in this organization. There is a great group of special people involved. Special thanks, of course, need to go out to Bill Miller and Frank Merrick and all of the First staff who worked so hard since St. Louis to insure we continue to have a quality competition. Thanks to Jon Dudas and the FIRST Board for supporting this investigation and their continued commitment to excellence.

Re: [FRC Blog] Einstein Report Released
 

I believe Mr. Dudas' letter, as well as other posts, are attempting to make it clear that the actions of the individual should in no way place a stigma against the team that the individual was formerly associated with.

Re: [FRC Blog] Einstein Report Released
 

After thoroughly reviewing the report, I have come to two conclusions:

1) FIRST did an incredible job of researching all of the factors that went into what happened on Einstein.

2) Any team, regardless of experience and ranking can have those simple little problems that could contribute to larger problems such as what happened on Einstein.

Regardless of whether or not the code and electrical problems were the root cause of the Einstein failures, each and every team should review this document and ask themselves whether or not they might have similar electrical/programming problems.

Loops without sleeps and bad crimps are problems that are easy to look for in your robot, but not necessarily easy to diagnose. If teams add these to their list of things to check on their robots during the build season, we can help ensure that robots are running to the best of their ability. Should any further malicious behavior take place, removing these problems as a potential source will assist in troubleshooting as well.

Re: [FRC Blog] Einstein Report Released
 

Where does his letter clearly state that?

Re: [FRC Blog] Einstein Report Released
 

While I still read, I think I should note to anyone who has hard feelings against 1114 or another successful team: what the heck are you gaining by beating someone else down? What would inspire you to go out of your way to see someone fail (the real kicker being you get nothing in return save for some personal satisfaction)? A bully beating up a bunch of kids in the FIRST playground is frankly depressing. I don't visit FIRST and FRC affiliated web sites to become depressed.

To make things a bit more light-hearted, I hope this act doesn't delay any development of SimPhone on Android. My Galaxy Nexus is a tame soul. It's letting me thumb through the report and push the link to some students on other teams.

Re: [FRC Blog] Einstein Report Released
 

^^Quoted for truth.

Furthermore, we should set the bar pretty high for characterizing an individual's intent to be "malicious".

The words in the report and Jon Dudas' letter are insufficient to support that conclusion.

We don't know the individual's motive or intent.

Re: [FRC Blog] Einstein Report Released
 

I agree completely. I can contemplate a few ways that someone might have been trying to actually help but picked a very bad way to achieve it.

Also, since there were issues outside of Einstein it sort of makes sense that there could be other ways this might happen that aren't quite so badly intentioned.

Re: [FRC Blog] Einstein Report Released
 

Oh man, I have never seen someone roasted so hard on CD.

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

In my short few years being around people, I've met a handful of people who are utterly brilliant but they have no social awareness and a lack of ability to see consequences. These people are nailed as socially awkward, but in their mind whatever they are doing, however they are doing it, is perfectly right.

Did this person interfere intentionally? Yes.
Do I believe that this person had malicious intent at heart? No.
Do I believe that the person has been adequately punished? Yes.

Additionally, I cannot even begin to imagine how team 1114 and their alliance partners currently feel, however, I believe that making a public apology is not the right path. The internet is the internet, and FIRST is FIRST. By publicly revealing himself/herself and the team, they leave themselves open to emotional, and possibly physical, harassment.

I am not condoning these actions, I am not condoning this person. However, without knowing all of the facts, I cannot support any action that would possibly bring this person any more harm.

- Sunny G.

Re: [FRC Blog] Einstein Report Released
 

I agree with you, but then also having these findings say that some of these teams lost connection because they were DDOS'ed and thats why they couldnt participate in matches, makes me say that the students, mentors, and parents of those teams deserve more than what this student has supposedly gone through. 1114 and 2056 specifically, because they were the 2 I know that said they were DDOS'ed, should feel the worst because their teams worked hard all season to accomplish what they did and got to the level they did this season and then had it swiped away by someone that had nothing to do with the match.

Re: [FRC Blog] Einstein Report Released
 

For all we know the person who they know had knowledge of it was just the tip of the iceberg. They could have told someone else that thought it was funny and did it to. They could have found this out from someone else that was also doing it.

There could also have been some other factor at work and let's hope it's one of those acknowledged within the report because a large amount of cost and effort went into this after the fact.

Re: [FRC Blog] Einstein Report Released
 

This reminds me of a story told to me by one of my university profs in networking.

Some friend of his found a flaw in the City of Toronto's (IIRC, big metropolitan centre anyway) traffic light control system. Tried to tell city council and they ignored him, blew him off. So he turned every traffic light red in the city for several minutes. When they came after him? He said "Be thankful I didn't turn them green."

Re: [FRC Blog] Einstein Report Released
 

After reading the report, I think I could say that I now support FIRST's claims of "Its not us, its you" in regards to comms loss at the regionals this year. If 5 of the 12 Einstein teams could have electrical/programming issues, I can see it being a larger problem then I thought before. Even though most of the errors were silly mistakes that wouldn't effect comss, imagine all the other teams that could make silly mistakes that would. The control system is fine, if put together correctly. More documentation is all you could really do about that, but teams have to read it. Now the radio, on the other hand..


I'm impressed with the detailed report and the handling of the situation by FIRST. Can't wait to get my hands on the FMS white paper!

As to the "interferer", I see his actions "accidentally intentional". From my understanding of the report, he simply tried to connect to the robot and provided an incorrect network password. I've seen robot networks on my mobile device many times, and tried connecting to it in the shop out of curiosity. Nothing really happened in that environment, but I'd give him the benefit of the doubt and call it a case of curiosity killed the cat. After all he did come forward.

Re: [FRC Blog] Einstein Report Released
 

I'm going to keep my personal feelings on the shelf for a while. I need to give it some time before I address more... unsavory... aspects of the situation.

But, what I am overjoyed to say, is FIRST did an amazing job at covering the situation. When I opened the document, I expected a 4/5-ish page summary report of what FIRST had been doing with the Einstein teams the past few weeks. I was pleasantly surprised to find an extremely long, fully detailed report of EVERY test and analysis run by FIRST. Bravo, FIRST. Bravo. You owe us nothing. Yet you went through everything for us. You guys rock!

I'll leave it with this. Do we really need a pound of flesh?
Just sayin'.

Re: [FRC Blog] Einstein Report Released
 

Every year someone that worked very hard has ultimate victory just out of grasp as there is a certain amount of random risk in this competition. I don't think that these teams are going away entirely empty handed. Though I do respect that they got a raw deal.

We can't assume that the person involved is any more guilty than a hypothetical weird wiring issue on the field.

We don't know if they were the only one doing this at all.

We don't know if they fully understood what they were doing.

We don't know how it was if they were trying to fix it they were dismissed.

I can't feel the need to ruin what could be some person's life when these teams will continue and have more chances to make it that far.

Besides this is perhaps one of those stranger and more memorable moments where the memory of the participation is more valuable than the dusty trophy. I'm sure a great number of us won't soon forget what happened.

Re: [FRC Blog] Einstein Report Released
 

Imagine the chaos if FIRST allowed any random person to walk in and give their suggestions while trying to diagnose the issue during Einstein. They'd be flooded with every person out there who has a theory (take a look at the numerous Einstein threads filled with people bickering about what they think happened to get an idea of how many people that would be) to step in.

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match. If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to? Because of those facts, I think it's hard to say that the intent was not malicious.

My heart goes out to The Eh Team. Nobody doubts that they were an amazing alliance of 2 of the best teams (and one very promising up and comer). When people look back on Rebound Rumble, they will not only remember that 16, 25, and 180 won, but that 1114, 2056, 4334, 987, 233, 118, 548, 2194, and 207 were all outstanding teams whose robots excelled at the game.


In Jon Dudas' statement, he says:
Nothing about the other Einstein teams, though.

Re: [FRC Blog] Einstein Report Released
 

Im really really glad FIRST took their time to put together an awesome report! BUT.... after reading the whole things twice, I'll admit I don't quite have the technical experience to understand the report in its entirety.

Could someone summarize and explain the more detailed aspects of the Robot Testing and Failed Client Authentication Testing? Specifically, what intentional interference actually happened, how did it cause problems, and what are they planning to do to fix the issue?

Thanks!

Re: [FRC Blog] Einstein Report Released
 

Firstly, thanks to everyone in and out of FIRST who made this exhaustive testing and report possible. It is great to have such a thorough analysis of the forces at play on Einstein and the lengths they went to replicate on field conditions were extraordinary. I sincerely hope this leads to dramatically less communications faults at any event this year.

It's absolutely appalling that someone in FIRST would sabotage an alliance (and then some) by exploiting a security vulnerability. It's also appalling that it could be so simple to knock a robot out of commission in any FRC match since Week 4. Cisco's got some 'splainin' to do...

I'm having trouble processing the idea that someone could deliberately attack all of the robots on an alliance by exploiting a security vulnerability without "malicious intent". Perhaps they were trying to raise attention of an issue FIRST should have known about (just about the only scenario I can think of that would even resemble "good intent"). Einstein is by no means whatsoever the proper time and place to demonstrate this problem.

And does it even matter what his / her intent was? Are the affected teams supposed to feel better about being cheated out of a fair chance at victory because "oh, he / she had good intentions"?

Re: [FRC Blog] Einstein Report Released
 

As the report points out a not trivial amount of problems were actually on the robot. This would make it hard to see cause and effect while testing such an attack. You wouldn't know if the attack stopped the robot or the robot stopped for reasons you had nothing to do with. Just like the FTA and other FIRST folks wouldn't know if the robots were broken or someone was causing trouble.

Add to this the storm and the other distractions it's just a perfectly bad combination.

Re: [FRC Blog] Einstein Report Released
 

I know I need to let this all marinate, but if the cause is so noble there is a far better way to achieve the results without taking it out on an alliance that has quickly become a punching bag in darker corners of the community (even though a member of it is the newest team in the FRC HoF...)

:/ this is rough stuff

Re: [FRC Blog] Einstein Report Released
 

I think taking down Einstein is the equivalent of "turning the lights green" here though.

Re: [FRC Blog] Einstein Report Released
 

The poster he is referring to stated that non of the Einstein teams were involved, not none of the winning alliance.

Re: [FRC Blog] Einstein Report Released
 

S/he was asked to stop and did not comply. The person also attempted to inform officials about it, indicating (to me, at least) that s/he was aware that what they were doing could have been causing issues.

If it had been one match that this happened in and had the person put the phone away when asked, my feelings on this would be much different.

Re: [FRC Blog] Einstein Report Released
 

Then again putting the phone away doesn't mean much if you think about it. I can leave an application running with my phone screen off on any Android device.

So they put it in their pocket and it's still causing trouble. Then they take it out and it's still causing trouble. They are likely just as distracted as everyone else with the weather, the people watching, people's behavior near them, etc. It's really hard to say what people do when they put themselves in a bad spot like that. From the second they were asked to put it away they were in fact in a lot of trouble.

I don't disagree with your gut reaction. After all I had concerns about power issues before this all started and FIRST suggested we not test at Einstein those issues. As it turned out some of those issues existed who knows maybe they would have been found (I can't prove it either way). I even went so far as to ask the question in the official forum with regards to the championship.

However, no means no. We completed the testing of my little oscilloscopes for the power to the radio on off season events. We did it where it would do as little harm as possible to FIRST proper if something happened that was not expected. We did it with full knowledge of those who could be effected.

There was most definitely bad judgement at work with this person's choice of actions. They've made themselves a sitting duck for suspicion. It's highly unlikely we will ever know if they are the only source of the deauth issue.

The amateur way they handled themselves however, concerns me because usually where there's an amateur trouble maker that is easy to spot it is a distraction for someone that's not an amateur.

I mean not to start a witch hunt just to make the point. We live in a complicated world where rarely do specific effective security issues just magically appear to a single person. Like the systems themselves they are collaborative efforts. I have a hard time believing that a person so clearly asking to get caught just figured this all out on their own.

In a perverse way the drive to seek out the problem with the WiFi might have guided them to this issue via collaboration, but even then I doubt they crafted the attack and were ready to report it just in time of that one event.

I do computer security for a living in part so perhaps it's just in my nature to rarely assume the singular genius that hands themselves to me on the silver platter isn't covering for someone else.

Re: [FRC Blog] Einstein Report Released
 

Alexa, I think you are misinterpreting the previous posts. No one is suggesting that any random person should be allowed to walk in and give their suggestions while the FTAs are trying to diagnose a field issue.


The document says that there was a witness(es) whose description of a person using a phone matched the description of the person on the field earlier. Perhaps it was the same person.

We don't know.

Re: [FRC Blog] Einstein Report Released
 

FIRST officials spotted him twice:

Re: [FRC Blog] Einstein Report Released
 

+1 to this. The honorable way for the team involved to handle this, even if it was a rogue member of a team acting alone, would be to come forward and spare this community from speculation and suspicion for years to come. You're protecting yourselves at the expense of many other teams. It won't work.

Re: [FRC Blog] Einstein Report Released
 

My head spins there is so much to take in.

First, I join my voice to the chorus that is saddened and outraged by the behavior of this individual*. I don't suppose that this is the first malicious attack by an individual on another FIRST team or alliance but this attack is so brazen. I am sick. Really sick.

Second, I don't think that it is too much to expect the former team of this individual to apologize to the teams involved as well as the entire FIRST community. What is more, depending on the circumstances, I would think that it may be appropriate for them to assure the FIRST community that they are committed to Gracious Professionalism in its highest sense and that they are committed to rooting out anything within their team culture that may have contributed to fostering the behavior of this individual.

I don't want to start a rumor but does anyone else think that this must be related to the nonsense that went on at the Greater Toronto East Regional? If so, the Canadian FIRST community really has to work to lance this boil.

Third, I am really disturbed by this statement in the report:
This seems to imply that they don't have this now. Wow.

Fourth, they offer "Additional emphasis in training and documentation" as a mitigation for "Robot D-Link radio reboot due to power dip." I hate this proposal. It seems to me that either the robots must return to active duty (passing packets back and forth) in a handful of milliseconds OR the radios must work down to a voltage that is almost literally impossible to get to without tripping the breaker (say 1.5 volts or something ridiculous like that).

Fifth, it seems to me that FIRST (and the FMS) has one implied contract with the teams: We will get X% of your data packets from your Operator Interface to your Robot and vice versa within Y msec. In my view of the world, literally the SECOND the FMS breaks this contract for even one robot on the field, the match has to stop and be replayed. Period.

With the current system this vision cannot be fully implemented because of dead/disconnected batteries and other complications.

I will say that if FIRST had this as their standard, this attack would not have been prevented but the attacker's purpose would not have been accomplished, which may have kept the attacker from even trying.

Reading the report, it is hard to know if the FMS system actively monitors this contract... ...past is past. Going forward, I really think FIRST should implement such a system hosting another tournament.

Finally, I hope that the FIRST community can pull together as a result of these events. We must do our best make FIRST better because of it. I have faith in our ability to do so.

Regards,
Joe J.

*I say "individual" because I am not sure if there were two or one. The report seems to indicate two folks were involved but only one was banned for life so... ...perhaps I am misunderstanding the text.

Re: [FRC Blog] Einstein Report Released
 

While this is a terrible event (it's terrible, it happened, end of story), my concern is that field issues happened all around the world during regionals/districts. Was a similar event the cause of all of those disruptions? There are people in the world who sometimes make the wrong decision, but to have such a person at all of the events where connection issues were present doesn't seem right.

Re: [FRC Blog] Einstein Report Released
 

I think that FIRST did a great job at handling this situation. This is something that could have easily been ignored or swept under the rug. No one would have known. Instead, they put a lot of time and money into the investigation.

It is a shame that something like this would occur in a student-based organization but it is comforting to know that FIRST puts its participants and ethics first.

I am so proud to have been a part of such a great organization.

Re: [FRC Blog] Einstein Report Released
 

I disagree entirely. I don't believe anyone believes (or believed prior to Einstein) that the system is above flaw.

Take any system, no matter how well designed, and subject it to 60,000 ambitious folks all playing with it and see how secure it is.

This week's 'Yahoo' password hack displays just what happens when even the most competent network security is open for public interaction. Someone WILL find a way in. Google, Microsoft, and even the stock market have been subject to security invasions as well.

I hate to say it, but in this situation security through obscurity is FIRST's best bet. The entire system needs to be removed from the consumer electronics spectrum that all these common tools are designed to work with. I.e. - standard a/b/g/n wireless needs to disappear. If this does not change and go to a proprietary system, I will 100% guarantee you WILL see this happen again.

Re: [FRC Blog] Einstein Report Released
 

I actually would prefer this team remain anonymous. Reason is that the Team itself is not responsible (or so it seems). And what ever the mentor/student/parent did wont ruin the whole teams reputation. If the team is revealed the team will become ashamed for what another individual did and could possibly shut down. I would hate to see this happen. Its not the Team's fault and I would hate to see them suffer from it.

Just my $0.02 CAN

Re: [FRC Blog] Einstein Report Released
 

As much as I can, I use the concept of gracious professionalism as an internal yardstick. However, I have a really hard time believing anyone would think that the correct way of carrying yourself when in possession of an issue like this is to disrupt the climax of FIRST's largest, most-anticipated, most-covered event.

I believe there are far better ways to demonstrate these bugs. FIRST has held Beta Day in Manchester. If the FTA and some volunteering teams were on board, I'd be okay with someone demonstrating a novel flaw the night before SCRIW and forwarding the information to FIRST. And if you feel the urge to break something at Championship, why not practice matches with your own team?

The phrasing of Jon's email makes me believe the individual involved is (well, was) a mentor on a team. As the guy in denim says, we get the best of what we celebrate. There is no room for celebrating interference with any competitive FRC match, whether it's Einstein or Q12 at some Box-On-Wheels Extravaganza of a regional. There is plenty of room to celebrate mentors that discover field issues and disclose them responsibly.

Re: [FRC Blog] Einstein Report Released
 

http://www.chiefdelphi.com/forums/sh...&postcount=115

Re: [FRC Blog] Einstein Report Released
 

Just to be clear, you're not implying that anyone who has posted to this thread is suggesting that "celebration" is the appropriate response to what happened, correct?

Re: [FRC Blog] Einstein Report Released
 

Yes. Intent matters.

Re: [FRC Blog] Einstein Report Released
 

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people :) But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

Re: [FRC Blog] Einstein Report Released
 

There's a bug in the version of firmware used on the field access point in Week 4 and later. It manifests itself when a D-Link with Rev A hardware is connected to the access point. If a "rogue" client attempts to connect to the wireless network and provides an incorrect WPA security key, the wireless link between the robot and the access point is disrupted. A subsequent failed connection attempt by the rogue can restore the robot link.

The "intentional interference" was someone using a smart phone to try to connect to the network established for one of the teams playing a match and failing. That should have resulted in exactly zero effect on the networks, but the firmware bug caused that team's robot to lose its link to the field network.

As an immediate fix, they're reverting to a previous version of firmware for the access point. Testing showed that version not to be vulnerable to the Failed Client Authentication problem. The ultimate fix will involve fixing the bug, including a test for the problem when doing validation acceptance for new revisions of firmware, and putting specific features in place to detect and log such attempted connections.

Re: [FRC Blog] Einstein Report Released
 

If you did do that, then we wouldnt be having this conversation, which would in turn make you not go and invent a time machine. So seeing as that didnt happen, either you fail in doing so or you do invent a time machine but you get taken out by the duplicate paradox...of course if that happens, which I hope it doesnt, you in essence die the day you go back in time to stop yourself from screwing up at last year's championships. :p

Re: [FRC Blog] Einstein Report Released
 

In the grand scheme of things, it doesn't matter. The person was punished, and it appears that it's an acceptable method.

However, intention is something that turns a situation on its head. Intention is what's driving every single post about how appalled people are that this happened. We are assuming intent, and therefore we are writing posts filled with hate.

I will iterate that I cannot begin to imagine how the effected teams are feeling, and the goal of my post was not to curb their anger/disappointment, to that, I have no right. My goal was to simply promote an ounce of open minded-ness.

- Sunny G.

Re: [FRC Blog] Einstein Report Released
 

Since we're (mostly) engineers on CD, let's think of this as an engineering problem. There is a supplier-customer relationship where FIRST is the supplier (of the control system) and the teams are the customers (of the control system).

The customer must have requirements for the supplier. The basic requirements for an FRC control system by a team are:

-Must be easy to setup for a very inexperienced team. Even if the manuals are clear, complex systems are inherently easy to misconfigure even if the directions specify how to do it.

-Must have a boot time of no more than (x) seconds for the entire system. I would be willing to exempt the DS only because I know how slow Windows is. This time limit is to the advantage of teams who want to run quickly, and the field crew which can recover from errors more quickly.

-Must have a certain amount of minimum functionality that can be achieved with minimal computer skills by a team - I would include a requirement for a certain amount of default code, at least mapping JS axis/buttons to motors and solenoids, since I know how many teams previously relied on default code. For some reason, the current control system lacks this and I don't really like it.

-Must be protected from any interference - I believe this requirement previously existed but was not fully met. I will discuss this later.

-Must have certain safety checks implemented - Specifically loss of communication with driver station, crashing of team code, and network error. Currently the first two are not implemented well, the third is implemented with a packet CRC which is good.

Good requirements? I think this covers the basics.

Now how do we verify that we've set all of the requirements correctly? How do we set the exact parameters of the requirements (boot time, amount of sophistication of hacking attempts, default code, watchdog timers, etc.)? Area they created by the supplier arbitrarily, or does the supplier actually know what the proper values are from their experience in designing control systems? Does the supplier actually work with control systems like these, or are they guessing based on experience in another field (say, industrial controls vs automotive controls - Timing and power requirements are *very* different between these two).

There are a few key flaws in this system which are highlighted above. Who is making the requirements that define boot time, maximum time without communication before the robot disables itself, or the minimum sophistication of a hacker attempting to compromise the security system? Do the requirements even exist, or is it more of a best-attempt system for some of these variables?

Best-attempt parameters always ends up as a variable which can be compromised to meat any other goal, with no real loss to the supplier.

The next step in engineering is actually designing the system. So, let's skip that step and end up with what we have now.

Once the system is engineered, you have to verify that you have met all of your requirements. How do you test this? While nobody thought to test the case of 802.11 access requests stated in the document, and this is OK given the obscurity of the bug and it's existence in hardware from another supplier, I can guarantee that somebody thought about DOS attacks on 802.11 (if they didn't, they would have not come close to meeting the requirement set above). What is the solution to them? The AirTight device was clearly chosen to detect DOS attacks, without adequate testing to verify that it actually met the requirements.

I am not pointing blame to anyone or any company specific, as I know there are many people and companies involved in the design of the system, but it has issues on a whole that are nobody's fault.

This document clearly shows me why 802.11 is not used in critical applications in this way (open SSID broadcast especially). No offense to FIRST, but the choice for 802.11 is probably the largest design failure of the entire control system (including my rants on compile/download/boot times and stuff). This testing, specifically the individual who is anonymous, shows just how vulnerable the system is, and how little it is protected from any intention or unintentional interference from a device that everyone carries in their pocket.

As a final thought, if the issues are blamed on the teams in such large quantities, then something must be too complicated, ambiguous, or otherwise error-prone in the system (basically the system is too complicated if that many people can't get it setup correctly).

Someone earlier in the thread questioned the failure at GTR as possibly being a related event. There were more failures than just GTR (although it was the most publicized) that showed up in exactly the same way as what happened to Simbotics. Teams' radios were blamed by FTA's, and everybody lived unhappily because the radios did suck (they thought) and there was nothing they could do to change it. The FMS was "above blame" - Because the DS could not ping the teams radio but the other 5 DS's could ping the other 5 radios on the field, it must have been the fault of the team's radio. (on a related note, the FMS gathers very little logging data for itself, most of it is collected by the DS and forwarded to the FMS). The general idea is that, since the cause isn't definitely the field, it's probably the teams (especially since so many teams have so many issues with the radio power wiring) and then it becomes always the teams fault, and the FMS is never blamed for failure of communication.

Anyone want to suggest another air interface? I've been thinking about a few...

Re: [FRC Blog] Einstein Report Released
 

This (similar) post is in the thread with the FIRST letter and link, but i was requested to duplicate it here -


There will likely be several threads and posts about the report and the contents. Please remember this is a public forum, use caution and care in what you say and what you claim.

Also, please read the report before commenting on it. There is a lot of information, and many questions and comments may be answered in the report itself.

Regarding the report, this is a detailed summary of the fact-finding, the process, the testing and the results and conclusions found by a large, diverse team. You may or may not agree with all of the conclusions drawn, but there is a great deal we can all learn from the detail of the report.

We also owe a lot to the 12 teams involved and their level of participation at the FIRST weekend and data collection process.

Also, note the request from FIRST for input. If you have constructive input, please use the email address to provide that to FIRST.


* Addition -

Read the report and the detail for what it actually says, not for what you think it says. And, to keep the thinking clean, if someone make a conclusion or statement not supported by the report, then call them on it and clear it up. Some are doing this already and it keeps the conversations and conclusions accurate.