Re: [FRC Blog] Einstein Report Released
 

In case you interpreted that section in this way, the white paper isn't needed to document the system -- there are dozens of engineering requirements and development documents, but to publicly release a comprehensible description of the system so that all issues involving robots or field can be resolved more quickly, with less guesswork.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

As the report showed, there were certainly a lot of issues with communications, coming from a lot of sources. And those are just 12 of the best teams in the world. Team error likely supplied most of the connection troubles throughout the season, just as it was the root of many incidents on Einstein. I severely doubt this is a repeat offense from earlier in the season.


With that said, let's look at the big picture. Every coin has two sides, even this one. It's not even close to fair what this "individual" did to those teams, or even their opponents. It never will be. But trying to look backwards, like thinking of replaying those Einstein matches, wouldn't be fair to anybody. Instead, appreciate what Frank Merrick and the people in FIRST have done, making the very best out of a terrible situation, and look forward to the more promising future. This incident has spurred FIRST into looking intimately at how the FMS works, giving rise to the potential for positive change that can make the years to come better than today. As much as it should, nothing will change from the 2012 FRC World Championship. Celebrate all twelve teams who were crowned Division Champions, and have faith that 2013 and beyond will be the best FRC seasons of all.

And it appears that we will never know the motives, or even the intent, of the "individual", so let's not pile on or ask unreasonable questions of or about the individual or his/her team. Going through the rest of your life without the opportunity to be involved with FIRST is already a tall order for someone who probably loved FIRST and poured as much into it as all of you have, but made one horrible mistake to bring it crashing down.

Re: [FRC Blog] Einstein Report Released
 

I think Leeland's (and others') reasons for allowing the person(s) who interfered with the matches on Einstein to remain unnamed are spot on. Based on my past observations of how the Chief Delphi community has handled situations where someone behaves inappropriately, it is INEVITABLE that both the person(s) and their team(s) would be metaphorically crucified. The response could have unintended consequences to the person(s) emotional state. In addition, the team would likely be unable to recover from the blow to its reputation. This is because, in my experience, the Chief Delphi community DOES NOT separate the actions of an individual from the actions of the team. Often times, I see members state that "your words/actions reflect upon your team."

For this reason, I would understand if the person(s) who engaged in the interference declined to make their identity known.

Re: [FRC Blog] Einstein Report Released
 

On a different note, this is a brilliant example for teams of how to do Root Cause Analysis. It's also quite incredible to see what happens when you go over even the best robots with such a fine-toothed comb. Everyone can learn lessons from this document -- and even moreso when the FMS whitepaper is released! :)

Re: [FRC Blog] Einstein Report Released
 

Throughout the season, we saw a lot of posts and statements about "connection issues" at event and champs. Many people blamed FIRST through the entire season.

However, as this report shows, there are a LOT of issues that can affect a robot's ability to perform on the field, and many of them are caused (unknowingly) by the teams themselves. In the future, we should all keep in mind that these robots are very complicated machines, and there is rarely a single root cause for "connection issues".

FIRST did a great job with this report, and the sheer number of issues they discovered with the individual robots really shows how detailed they were in their investigation.

As for the individual who caused interference on Einstein... It's all been said at this point. The individual has been punished, and there isn't really anything else we can do about it. Part of Gracious Professionalism is not pointing fingers. When we find bugs at work, we don't ask "who wrote that section of code?" We ask "Who is the best, most knowledgeable person to work on fixing this bug?" As a community, lets move past the actions of the individual and show our support for the job FIRST has done.

Re: [FRC Blog] Einstein Report Released
 

I did exactly the same thing and spent a couple minutes thinking of just how awesome it would look to have towering Van de Graff generators over the old Manchester mill buildings... :D


This report is really awesome. Two thumbs up for FIRST because they really did their homework. Interesting a lot of the "Oh it can't possibly be team XXX" turned out to be not the case, and I think it will probably serve first well in the future as the community will probably not jump all over them at the first sign of trouble -- especially if high quality work like this can be expected in the future. It's also a side of engineering (failure analysis & technical report writing) that many FRC kids don't get to experience, so it's a interesting exercise from that point as well.

I'm just glad that the "hacker" is no longer in a position to ""inspire"" students.

Re: [FRC Blog] Einstein Report Released
 

It's is in my experience over 17 years extremely common for people to assume that the field can not be the source of a problem. Often they are correct but that only makes it more troubling when they are not.

No problem I do that every day literally. Only it's more than 60,000 people. I do computer security for a living as well as operating a few businesses that work with computing, electronics, and electrical. We have lots of security problems and we do our best to identify, qualify, quantify, document and offer resolution.

Sometimes we get solutions and sometimes we do not.
If we don't get resolution then we know where to look when the trouble starts.

It's off topic but that's a bad example. Anyone competent wouldn't store a password in clear text in a database with that sort of exposure to risk. You pad, hash and salt (and it's very simple there are existing tools to do this for you). They obviously left this old stuff laying around without regard for the SQL injection attack that is all the vogue for XSS these days. In point of fact we've been using this as a wonderful example of exactly why I have a policy document for the developers to avoid this exact attack vector (they are only very lucky that it wasn't a black hat that went after them quietly). They were also not very forthcoming about the possibility of the scope of the breach as they have a XSS single sign on they implement. Worse...some people think it was 'Yahoo Voice' that was breached but there's another Yahoo service 'Yahoo Voices' (that's right it's one letter off and the reporters who have to handle the announcement are not keeping it straight). It's an example of everything you don't do if you value your security or your business before, during and after a breach.

I grant you they have lots of other security issues at Yahoo right now that I am well aware of as are plenty of others. Surely they are not the only company that fails to be vigilant or gets utterly complacent. I'm sure someone figured they were saving a dollar (and maybe they did).

However, not all breaches are equal. The more people know about a problem the more silly you will feel when you get nailed for it. FIRST's deauth vector is not new, Hack-A-Day exposed this very publicly last year and other sites well before that. All that was required to breach this? Download code.

Perhaps the most effective hack is not a hack at all. Social engineering is the easiest and most effective hack because it hacks people. However, you don't differentiate you consider them all the same. Social engineering hacks are also why what you write next will not be nearly effective as you think:

Actually it doesn't matter what spectrum you use or how obscure. It's radio and it can be blocked cheaply and easily (though obviously illegally...but they gotta find you and prove it). As long as it's wireless denial of service will always be possible if you're willing to take enough risk as the attacker.

Obviously a band less frequently used will make it more obvious what you are doing. However once you commit to those frequencies without recourse they could hold you hostage long enough that the cost to continue will be extreme.

I don't actually disagree that they should move some of this from the bands where people accidentally could interfere with phones and such. I just don't think it matters as long as the field aspect is assured. So in that regard I think the field comm. specific stuff should be put somewhere and let anyone use WiFi for whatever they like. Let the users deal with the security issues, finding channels, and if you like give them a solution that ought to work in that regard but get out of the business of letting student written robot code interfere in field comms. The fields comm. stuff is generally unique to the competitions anyway, outside of the competitions WiFi is plenty workable.

Re: [FRC Blog] Einstein Report Released
 

The attacker's motives don't strike me as dark and mysterious. The report specifically states that the individual contacted FIRST and explained what they had done. If they hadn't, we would probably still be speculating. Were their actions in St. Louis wrong, foolish, and harmful? Yes. But I hesitate to say they were malicious.

Imagine this: You're sitting in the stands at your regional, watching a match, and just out of curiosity, you try to connect to the FMS from your phone. The phone asks for the WPA key. "Of course," you think to yourself and give up on this little experiment. But you punch in a random password anyway, and to your horror, the robot on the field suddenly stops.

You try to tell the FTA, but he brushes you off. Lots of robots die for lots of reasons, and there's absolutely no reason for a failed authentication attempt to take out a connection.

Now what?

From reading ChiefDelphi, it's clear to you that this is probably happening all over the country. FIRST HQ seems to be ignoring the control system issues and is blaming the teams instead, and it makes you angry. (Go back and read some threads from March - this sentiment abounds!) Championships are going to be a mess. It's almost certain that others are going to discover the issue, and they will probably use it in a malicious way to gain an advantage in the competition.

So, in anger and wannabe-heroism, you do the only thing you can think of to get FIRST to listen...

Was this the wrong way to address the issue? Absolutely. But given the openness, sincerity, and determination we've seen from FIRST in recent months, I wonder if the attacker succeeded.

This investigation uncovered a treasure trove of software bugs and electromechanical faults, and I hope we will all build better robots next year because of it. It showed that problems, both incredibly complex and completely stupid, will happen to all of us. Major thanks to FIRST for such a thorough investigation, and I'm looking forward to the improvements next season!

Re: [FRC Blog] Einstein Report Released
 

I'll just leave this here.
Anyway, this will be my last post on the matter. The intentions can never truly be known.

Re: [FRC Blog] Einstein Report Released
 

He was speculating as to how he / she discovered the bug - not how he / she executed it in St. Louis.

Re: [FRC Blog] Einstein Report Released
 

Pretty close to malice.

That situation is of course a hypothetical one, but we can't justify or glorify the hackers actions because of the "silver lining" this investigation was.

Re: [FRC Blog] Einstein Report Released
 

This is going to sound strange but scouting app?

After all even the Einstein teams will compete off season.

Several teams have applications like this in the Google Play store that anyone can download.

Re: [FRC Blog] Einstein Report Released
 

That have live match information? Not even the FIRST database has live match-by-match data. It couldnt be anything else other than the FMS system this person was looking at.

Re: [FRC Blog] Einstein Report Released
 

The information contained in the report and the information omitted from the report was selected for a reason. Attempting to read between the lines will only grant speculative conclusions. Wild speculation is exactly what this report was attempting to mitigate.

The "Root Cause Conclusions" table on page 20 of the report lists fifteen instances of command response failures. Six of those instances have nothing to do with the "Failed Client Authentication" issue. Only one of the fifteen instances is "confirmed" to be because of the "Failed Client Authentication" issue. Eight matches were ran on Einstein, each with six teams participating. That's 48 potential opportunities for command response failures. Setting aside the FCA issue, 12.5-14.6% (depending on 2056's root cause in SF2-1R) of the opportunities still manifested command response failures.

Regardless of whether or not the root cause was the fault of the field or the team, a system with a 12.5% failure rate among the elite participants at the end of the season is simply not acceptable. This report was not about the practice day at an early season regional, but the finals of the championship event. A vast majority of this report is not focused on the Failed Client Authentication issue, but the numerous other potential points of failure in the system both on the robot and FMS sides of the equation. I'm glad FIRST is taking steps to try and improve this system and remedy the issues highlighted in this report, and this is far more important to me than debating the motives and proper punishment of an individual.

Re: [FRC Blog] Einstein Report Released
 

Great report and worth waiting a few weeks for. It's easy to become cynical about the organization, but this is an example of FIRST doing things right.

Quick thoughts:

• Disappointed but not really surprised that there was intentional interference. It's been a huge risk and getting worse now that many participants are carrying at least one 801.11n capable device in their pocket.
  
• Surprised that even the top-tier teams had so many electrical and software problems. I've seen a variety of issues among the regular teams at regionals and this does all suggest that control system support is inadequate even now that we're at the end of it's 4th season.
  
• The overall programming model is not well understood, as evidenced by the multiple issues with *Continuous methods. Better documentation here will be very welcome.
  
• Similarly, two thumbs up for the promised FMS whitepaper.
  
• Unfortunately the "individual" will become known. Not that many people are on/next to Einstein during the finals, and this community is too curious/dogged for the information to remain hidden.