[FRC Blog] Einstein Report Released [Archive]

techhelpbb

13-07-2012, 17:47

I'm happy to see that the electronic power supply issues were considered and annotated. All the testing I did at off season events showed few and far between issues related to the AP/router power supply with some issues with the AP/router power supply (from the battery to the radio power input) taking a bit longer than might be a good idea to reach full output regulation from cold power up (how sensitive the AP/Router is to this is variable and some units might be effected). The time the AP/router supply takes to reach full regulation voltage may leave room for the routers to come up in strange states. This makes sense, all the teams suggesting they fixed their problems when they powered down the AP/router a second time and then back on again after powering up the robot when they initially got on the field (how often it happens though would be quite hard to determine).

The person attempting to connect to the field network is bad news and as long as critical field functions are connected to a public common network FIRST will run this risk. Lucky for everyone this was done line of sight to the effect. Frankly the attack (lack of common sense/failure to communicate intentions) in question could easily have been done any number of clandestine ways that would not have been noticed (I am not going to list them out of concern that someone will try them). Unfortunately it's really easy for someone to create something that will attempt to connect to the field network while aggressively hunting for Internet connections.

I see that they considered the antennas for the field and a few placements of the field equipment. I would have thought they would have tested that further with the robot side equipment power measurements as both sides transmit and receive but apparently that did not happen. Suggestions on how to best optimize the range of the KOP standard robot mounted AP I should hope will find their way into next season.

Overall, I'm satisfied that they've done all they can with what they have as far as a test is concerned. I'm not sure I'm convinced that this one person was the cause of so many headaches however. Einstein was hardly the first time connectivity issues surfaced that were not readily explainable by power supplies issues or programming.

My thanks to FIRST and all those that have worked so hard on trying to make sure this does not happen again.

linuxboy

13-07-2012, 17:48

Hmmm. Intentional interference... sounds like FRC is really starting to look like a "varsity" sport.

Aside from Einstein having issues there were tons of radio and cRIO issues at regionals. It really is time for FIRST to look at a new system.

I believe Bill mentioned that the NI contract will be up for the 2015 season.

- Oliver

PayneTrain

13-07-2012, 17:48

The organization did what they were expected to by members of the FRC community and more, even with the news arising that intentional interference took place. my eyes really widened when I read that. I'll finish it after I let that thought sink in.

It would have been a poor option to take away the award from the kids and their organizations that support them, but it's great that they extended the berths to all teams on Einstein and paid for the teams to register in 2013.

davidthefat

13-07-2012, 17:50

Akash Rastogi

13-07-2012, 17:51

I think it is important to remind readers to not be too shocked and awed by the fact that interference happened; I think it should be viewed as another thing that is potentially a problem with the control system.

My point being; don't let this news be the face of the issues with the control system. The fact that these elite teams were able to have such problems with the system leads me to say that perhaps this is not the ideal system for FRC.

This report is great and revealed many problems teams may have dealt with all season.

+0.02

ttldomination

13-07-2012, 17:55

After reading through this, I believe FIRST should either replay the matches or declare no champions this year and allow all 12 teams free payment for next year's championship.

I don't know if you are joking or not, but the ramifications of this would be ugly, at best. The current step was the correct one.

I believe Bill mentioned that the NI contract will be up for the 2015 season.

- Oliver

Meh, teams have thousands of dollars of control system laying around. If we must, then I suppose we'll go to a new system, but only if we really have to.

Time to read the detailed report,
Sunny G.

bearbot

13-07-2012, 17:55

WOW thanks you to FIRST for releasing this out to everybody I appauled that it hopefully wont happen again.We had major issues with the FMS @ Cheaspeake regional and saw they were used on Einsten field to.Whish FIRST would have done something to the fields that the electronics came from to instead of just from the Einsten matches

Racer26

13-07-2012, 18:02

Wow. Intentional interference. Shame they decided to let the results stand after that. I'm sure 1114/2056/4334 are left with a rather sour taste in their mouths.

I read the whole report, and while a number of possible issues were identified for the various teams, only 118's dropouts were really confirmed to be caused by team equipment/configuration.

I will say that for the FCA attacks on 1114/2056/4334 to be intentional, someone would have had to have figured out that the FCA vulnerability existed, and based on the way the report talks of it, it seems that this only existed from Wk4->Championship, and only on robots with rev A hardware. Curious, indeed.

I still stand by my original assessment from April that the proper way for FIRST to have handled Einstein is to award all 12 teams 2012 FRC Champions, in lieu of being able to play a fair set of matches.

Joon Park

13-07-2012, 18:02

I am just as shocked about the intentional interference as everyone else is, but I really would like to extend sincere thanks to the twelve teams, the volunteers, the experts, and most importantly to FIRST and FRC for taking the time to deal with this in such a professional and thorough way.

JB987

13-07-2012, 18:05

ಠ_ಠ So, asides from the intentional foul play, it seems as if their was a lot of errors on the teams themselves. Seems like 987's problem was that there was a deadlock with their software. So, that's a shame.

David,
If you read the full report carefully and pay close attention to the following:

987 – While testing of this robot revealed programming issues that could cause higher than normal trip
times, or the cRIO controller user code to lock up, none of these issues were found to cause a command response failure. Which was the cause of our "dead" time on the field). The duration of these losses were too short to be attributed to a cRIO or robot radio
reboot, but fit well with the symptoms of failed client authentication.

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

davidthefat

13-07-2012, 18:08

David,
If you read the full report carefully and pay close attention to the following:

987 – While testing of this robot revealed programming issues that could cause higher than normal trip
times, or the cRIO controller user code to lock up, none of these issues were found to cause a command response failure. Which was the cause of our "dead" time on the field). The duration of these losses were too short to be attributed to a cRIO or robot radio
reboot, but fit well with the symptoms of failed client authentication.

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

Yes, I admit, I have not read carefully enough. Still, that's a shame IMHO. A deadlock anywhere is not good.

Steven Donow

13-07-2012, 18:09

Wow. Intentional interference. Shame they decided to let the results stand after that.

Barry Bonds used steroids, however in the record books he is still listed as hitting the most amount of homeruns.

Just thought I'd throw that in for comparison. Also, it isn't known(well, to the general public, i'd assume those involved do know) whether or not the interferer was directly targeting a specific team; just that he was only SEEN targeting 2056 directly.

IMO intentional interference being reasoning to change the results is an invalid point. As stated, FIRST is now more like many varsity sports :D

Also of interest was that it seems like 118's issue was also code and that no team on Einstein was using Labview. For reference, do 1717 or 469(had comm issues in their semis) use labview?

Racer26

13-07-2012, 18:11

R.C.

13-07-2012, 18:11

For reference, do 1717 or 469(had comm issues in their semis) use labview?

1717 Did not use labview for sure.

Worth noting: I expect the FRC Community at large to be wholly unsatisfied with HQ's decision not to replay the matches (and/or declare all 12 teams Champions in lieu), given the circumstance of intentional foul play.

I don't think you can blame FIRST for not replaying the matches... Replaying Einstein 2012 would be asking for trouble. Teams or the community shouldn't be unsatisfied. The alliance of 25/180/16 were darn solid. 25/180 hit their shots and could triple. 16 was by far too freakin amazing at stealing/feeding. And the alliance had a really solid auton/endgame.

Teams and the community should be stoked that FIRST took a giant step forward and are working towards figuring out issues with the control system. They even published a very nice paper. Kudos to FIRST.

-RC

Radical Pi

13-07-2012, 18:17

I will say that for the FCA attacks on 1114/2056/4334 to be intentional, someone would have had to have figured out that the FCA vulnerability existed, and based on the way the report talks of it, it seems that this only existed from Wk4->Championship, and only on robots with rev A hardware. Curious, indeed.

I have a feeling that while there was intentional FCA going on, there's a large potential for unintentional FCA as well, which they didn't mention whether or not had been tested. Many teams configure their practice network at the shop with the same SSIDs that the field uses, and if they used a different security key, any phones/tablets/laptops that happened to be active during the games and had been connected to the network at home could have been attempting connections in the background. I actually caught our team's programming laptop trying this once during week 2.

Also, I was surprised to notice this year that SSID broadcast was enabled on the field APs, making the team networks show up on every laptop in range. I don't remember it being that way last year.

On a somewhat related note, what would people think about having a CSA check over code as part of inspection to counter the kind of user code issues that were seen in the report. It wouldn't have to be a pass/fail kind of thing, but having a more well-trained pair of eyes to provide suggestions to the team and something for field staff to go on when a robot misbehaves would probably go a long way

Anyways, excellent report by FIRST, and I think bandwidth caps are a great idea. I hope they don't take the partial involvement of the BeagleBone as a reason to ban non-KOP electronics though. They really open up new possibilities for teams.

EDIT:
Also of interest was that it seems like 118's issue was also code and that no team on Einstein was using Labview. For reference, do 1717 or 469(had comm issues in their semis) use labview?
I don't know about those two, but after those semis 68 had unexplained comms issues in the finals, and they do use labview

whattsheorder

13-07-2012, 18:19

Impressed by FIRST's response, my faith in FIRST has been restored. So ridiculous what the hacker did though. I'm curious as to what his relations are to FIRST/any teams..

Basel A

13-07-2012, 18:24

connor.worley

13-07-2012, 18:33

I am definitely happy to see first taking steps to resolve the issues they discovered, especially with better documentation for advanced coding, the investigation of a new radio, and a fix for the NetworkTables issue.

davidthefat

13-07-2012, 18:36

Wait, isn't this the reason why you debug and profile your code?

connor.worley

13-07-2012, 18:37

Wait, isn't this the reason why you debug and profile your code?

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

R.C.

13-07-2012, 18:38

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

Agreed, along with the limited time and constant updates.

-RC

JB987

13-07-2012, 18:38

While I don't think there's a strong case for replay or any change to the officially announced results of the 2012 FRC Championship, I do think it's worth noting that prior to their being knocked out, the Archimedes alliance was the only one suffering problems likely caused by FCA. If the interferer had an agenda, it seems that 1114, 2056, and 4334 was their target.

You will notice in the report that it also seems like 987 suffered the same problem in Final 1 and 2...

Ross3098

13-07-2012, 18:38

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

I cant help but believe that the Curie Curse had something to do with the Curie alliance in the finals.... :ahh:

davidthefat

13-07-2012, 18:42

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

Well, I would disagree. From the language of the descriptions, it seems as if the problems would arise from general use. Sure, you'll definitely have to try hard to get certain conditions, but debugging and profiling your usual usage conditions is enough IMHO. Like the 100% CPU usage should have easily been caught.

edit: perhaps, it's just me. I like spending time with my robot.

Nate Laverdure

13-07-2012, 18:44

If I were the boss, I wouldn't have released this report on Friday the 13th.

Basel A

13-07-2012, 18:44

You will notice in the report that it also seems like 987 suffered the same problem in Final 1 and 2...

True, but 987 didn't have those problems in the semifinals, suggesting that the Archimedes alliance was the higher-priority target. Of course, you're right, the fact that 987 suffered the same problem means that you were probably targeted as well. While it's regrettable that this happened to anyone, but it seems fairly clear that there was a pattern to the chaos which began with 1114, 2056, and 4334's issues.

Jon Jack

13-07-2012, 18:48

I think FIRST has handled the situation well so far. FIRST could have easily turned a blind eye and tried to sweep the problem under the rug. What would replaying the 2012 championship matches really solve? Nothing. In my opinion, giving the 12 teams automatic invites to the 2013 Championship as well as waiving their first play registration is enough consolation.

there's a large potential for unintentional FCA as well, which they didn't mention whether or not had been tested. Many teams configure their practice network at the shop with the same SSIDs that the field uses, and if they used a different security key, any phones/tablets/laptops that happened to be active during the games and had been connected to the network at home could have been attempting connections in the background. I actually caught our team's programming laptop trying this once during week 2

The issue isn't just trying to connect to the network, the issue is trying to connect to the network - while entering an incorrect WPA key:

During the Post-Championship field testing an attempt was made to connect to one of the team SSIDs set up on the field network, but the WPA key was entered incorrectly. This was observed to sever the communication
between the driver station and robot associated with that SSID. After further testing, a link between failed authentication attempts and a disruption of the communication between a driver station and robot was confirmed, though not every failed authentication attempt resulted in a communication disruption.

Lets hope that FIRST continues to learn from Einstein 2012 and makes changes to the control system to prevent this from happening at future events.

apalrd

13-07-2012, 18:50

There were several things I got out of this paper, especially as an engineer working on engine controllers:

-The Smart Dashboard had a bug which was exploited which caused a deadlock. While all software has bugs, it should also be tolerant of failure, meaning the rest of the system should have been designed to operate (possibly in limited quantity).

-The Smart Dashboard was mentioned numerous times relating to increased network load, especially the funny 1-byte packets.

-The VxWorks operating system handling of the packet buffer seems exceedingly poor. Many other forms of communication (e.g. some CAN stacks) dump old packets with the same ID when they are added to the buffer, this seems like the right move (at least on UDP).

-The boot time of the cRio was mentioned to be 24s minimum.
--I am currently working with an engine controller that can reboot the application software fast enough to not stall the (Diesel) engine while it is running.

-The nature of 802.11 makes it a poor choice for this kind of wireless communication.

I will not comment on anything else.

Hjelstrom

13-07-2012, 18:53

Yes, I admit, I have not read carefully enough. Still, that's a shame IMHO. A deadlock anywhere is not good.

David, the deadlock is within the WPILibrary and we have been helping them resolve it. In fact, just today we were at the shop with our programmer Brandon as he finished up test programs for FIRST to use to reproduce two bugs we encountered in the smartdashboard support code this season.

These bugs did not affect our robot on Einstein or in any match all year because we avoided them. Working towards getting them fixed for next year is just a side benefit of the New Hampshire meeting.

Alexa Stott

13-07-2012, 19:00

Well, I would disagree. From the language of the descriptions, it seems as if the problems would arise from general use. Sure, you'll definitely have to try hard to get certain conditions, but debugging and profiling your usual usage conditions is enough IMHO. Like the 100% CPU usage should have easily been caught.

edit: perhaps, it's just me. I like spending time with my robot.

If you read the document carefully, it states:

While this may have impacted the performance of the cRIO, it is unlikely that it was the source of 1114’s command response failures on Einstein.

[...]

No issues were found during these tests that could explain the connection problems seen on Einstein by 1114.

Later on, it states that the likely cause of 1114's issues was FCA.

Also:
It is unlikely that the code issues found on 4334’s robot relating to the 100% cRIO CPU usage would be able to cause a complete command response failure for the duration experienced in the initial playing of Semifinal 2-1.

The 100% CPU usage was not found to be the cause of their issues so debugging that issue probably would not have helped (much).

I really don't think many of the issues described in the document can be attributed to teams not debugging carefully enough.

jason701802

13-07-2012, 19:01

I think FIRST has handled the situation well so far. FIRST could have easily turned a blind eye and tried to sweep the problem under the rug.

You mean like they have been doing since they introduced this system? This report is several years too late, and even now it fails to address many issues. It is a shame that this report does nothing to address the issues at have been occurring at regionals since the introduction of this system.

connor.worley

13-07-2012, 19:04

You mean like they have been doing since they introduced this system? This report is several years too late, and even now it fails to address many issues. It is a shame that this report does nothing to address the issues at have been occurring at regionals since the introduction of this system.

Issues in years prior may have had nothing to do with the deauth attack- only one radio and one AP that they tested were vulnerable. Older radios may not be vulnerable. I don't know if the AP has been changed.

jason701802

13-07-2012, 19:09

Issues in years prior may have had nothing to do with the deauth attack- only one radio and one AP that they tested were vulnerable, the ones used this year.

And that is precisely why this report fails at its root. I care very little about the problems that affected 12 teams when there are other problems with the control system that affect 100s of other teams.

Jibsy

13-07-2012, 19:10

If I were the boss, I wouldn't have released this report on Friday the 13th.

This made me laugh in the midst of being upset about the news. Thanks for that haha.

jblay

13-07-2012, 19:13

While I don't think there's a strong case for replay or any change to the officially announced results of the 2012 FRC Championship, I do think it's worth noting that prior to their being knocked out, the Archimedes alliance was the only one suffering problems likely caused by FCA. If the interferer had an agenda, it seems that 1114, 2056, and 4334 was the primary target.

I think this thought process is very dangerous, before you know it you are pointing the finger at certain teams that were on Einstein. It could have been so many iterations of people and pointing the finger at someone specifically the wrong someone is not what FIRST is about.

KrazyCarl92

13-07-2012, 19:17

If you read the document carefully, it states:

Later on, it states that the likely cause of 1114's issues was a sensitive main breaker.

Based on these results, it is unlikely that the sensitive main breaker would have caused command response failures during the actual Einstein matches.

It says the exact opposite of that.

Alexa Stott

13-07-2012, 19:20

It says the exact opposite of that.

Yeah, my bad. I fixed my post. :)

Karthik

13-07-2012, 19:21

FIRST Hall of Fame Team 1114, Simbotics, would like to thank FIRST for thoroughly investigating, addressing and documenting the robot failures that took place on the Einstein Field at the 2012 FIRST World Championship. We would especially like to thank Frank Merrick and his staff for constantly communicating with us through the process.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

Billfred

13-07-2012, 19:26

When I saw the email blast come through on my phone, I seriously sat in my car for 20 minutes or so reading the PDF. My thoughts:

1) I applaud the team for being this thorough in their methods and sharing all the steps they took (and for ruling out things we would normally hold up as the cause).
2) I believe FIRST has done right by the teams involved. Nothing is gained by replaying the matches.
3) I thank the teams for checking their own systems and code to discover issues of their own.
4) Shame on the individual who attempted this stunt. It hurts that FIRST has to have its answer to baseball's permanently ineligible list (http://en.wikipedia.org/wiki/List_of_people_banned_from_Major_League_Baseball). We should all expect a higher standard.
5) I hope some element of this process is framed and put somewhere fairly prominent in FIRST Place. I think this entire story contains elements of the FIRST experience both at its worst (see point 4) and at its best (see point 3). Much can be learned from both.

CalTran

13-07-2012, 19:27

That was one well written report, and certainly insightful to the whole FMS process. It gave our team a thing or two to look out for in the up coming seasons, as well as possible things to look immediately at for debugging.

As far as the repeated notions of a "tainted" win and questions of replay or total recall, I believe that is out of the question. 16, 25, and 180 won the 2012 FIRST Robotics Competition World Championship, and should not be forced to defend their title.

dodar

13-07-2012, 19:27

FIRST Hall of Fame Team 1114, Simbotics, would like to thank FIRST for thoroughly investigating, addressing and documenting the robot failures that took place on the Einstein at 2012 FIRST World Championship. We would especially like to thank Frank Merrick and his staff for constantly communicating with us through the process.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

Even though I think that the teams that had this happen to them deserve to know who did it and why, I foresee what you are asking turing into a similiar mindsight as the Jerry Sandusky case. That is, that the perpetrator did it full knowing what he was doing and what the consequences could be and forgoing the effect it would have on others, but it in turn not only effected him but also Penn State; the point im getting at is, putting forth the person and team publicly would do far greater harm to the team and their image amongst their peers and the FIRST community. I think a much better solution to what Karthik wants, and is justified in wanting, is to have all the parties involved meet so that everyone can be satisfied as to reasons why, punishment, etc... But I do beleive that this shouldnt be known to the masses.

techhelpbb

13-07-2012, 19:40

The larger issue than who did this is how was the system allowed to get to the point that it was possible at all. Let's worry about all the other D.O.S. (denial of service) attacks we have yet to find.

Clearly changes need to be made. It took extraordinary effort on the part of too many people to resolve even these issues to this point.

It still goes back to the assumption that the system is above flaw and that assumption being incorrect. In this case the system has a security issue and an active exploiter. Take if from me: you can look for and fix security issues before they get exploited as best you can or you can wait until they cost you reputation, resources and opportunity. Had they even profiled the issue before hand they could have dramatically reduced the chaos after the fact (if you don't fix it at least acknowledge it exists at the remediation level).

FIRST needs to consider a secondary channel in their control system if they can't more fully profile something like the WiFi system they have now. It's the logical alternative to pumping all the data into a single publicly exposed communication system. FIRST did have the AirTight equipment on site but clearly that alone wasn't sufficient to keep a lid on this issue. Additionally the FIRST report oddly doesn't discuss that the AirTight hardware did not produce for them sufficient warning of this issue or whether they consider that something they need to pursue (it was the assumption that this additional monitoring was sufficient to keep intentional interference from happening at all).

I give FIRST credit for the heavy work they put into this. I'm just not convinced that this won't happen again if someone intends to deny service to the field. Nothing I've seen in the recommendations will stop it. I can think of 3 ways right now that if I wanted to render all the field robots dysfunctional I could and it would be nearly impossible for them to discover it. Even if the sudden failure was recognized as intentional disruption the cost in resources to weather such interference is unacceptably high. A better solution needs to be found for this.

Alexa Stott

13-07-2012, 19:49

Even though I think that the teams that had this happen to them deserve to know who did it and why, I foresee what you are asking turing into a similiar mindsight as the Jerry Sandusky case. That is, that the perpetrator did it full knowing what he was doing and what the consequences could be and forgoing the effect it would have on others, but it in turn not only effected him but also Penn State; the point im getting at is, putting forth the person and team publicly would do far greater harm to the team and their image amongst their peers and the FIRST community. I think a much better solution to what Karthik wants, and is justified in wanting, is to have all the parties involved meet so that everyone can be satisfied as to reasons why, punishment, etc... But I do beleive that this shouldnt be known to the masses.

The PSU case is different-many university officials actively participated in a cover up. I don't think any comparisons should be drawn between the two in any case, given the difference in severity.

kjohnson

13-07-2012, 20:02

Additionally the FIRST report oddly doesn't discuss that the AirTight hardware did not produce for them sufficient warning of this issue or whether they consider that something they need to pursue (it was the assumption that this additional monitoring was sufficient to keep intentional interference from happening at all).

I know the report is long, but I urge everyone who wants to make a statement about the report read the entire report first. There have already been multiple posts in this thread containing false assumptions.

...
The AirTight system successfully detected and flagged this test as a “Denial of Service Attack” event.
...
The next tests involved targeting the attack at the specific client (the robot radio) and throttling the rate at which the tool sent the de-authentication packets. These attacks were also successful at disrupting the communication, but were able to elude detection by the AirTight sensor. Further investigation into the thresholds for classifying an event of this type in the AirTight system revealed a minimum of 8 directed or 4 broadcast de-authentication packets per second for a minimum of 90 seconds must be detected for the AirTight system to flag and classify a denial of service event. FIRST has determined that these thresholds provide insufficient protection for the FRC wireless use case.

A number of solutions to the network susceptibility to a “De-Auth Flood” attack will be investigated including working with AirTight to modify thresholds for detection, implementing custom capture and detection of these attacks, and changing radios to a custom solution which is more resistant to these attacks.

bardd

13-07-2012, 20:09

Worth noting: I expect the FRC Community at large to be wholly unsatisfied with HQ's decision not to replay the matches (and/or declare all 12 teams Champions in lieu), given the circumstance of intentional foul play.

If one of the winning teams had to do with this interference, or even any of the teams who played in Einstien, I would have agreed (of cource, not giving the interfering team any title). Since Jon Dudas's letter clearly states none of these teams were involved, I think FIRST made the right decision.

techhelpbb

13-07-2012, 20:13

I know the report is long, but I urge everyone who wants to make a statement about the report read the entire report first. There have already been multiple posts in this thread containing false assumptions.

I saw this already as I've read it three times. I consider my statement correct.

They assumed that the AirTight system was going to trap such attempts and they assumed wrongly.

The point remains that it's highly likely that there are many other things that AirTight won't trap (2 other that I am acutely aware of) that are not already discussed in this report.

In short, yes you can read this that I limit my concerns to deauth but frankly the solution isn't to fix one problem in AirTight and think that it's now fool proof security. That's the sort of thinking that created the problem.

They need to completely reconsider how they transit the really important traffic. Custom solutions in their context could mean anything (including loading existing hardware with DD-WRT or OpenWRT which itself is not free of exploit).

The long term risk is that all the focus sits on this particular vector of attack and polite offloading of all security concerns to AirTight continues to leave exposed other vectors of attack. It's not about blame at this point. FIRST has gotten more blame than they deserve in some ways and AirTight doesn't make robot WiFi security products specifically for FIRST. It is just about suggesting that AirTight has only that issue which is wrong and is what this report basically does. It's not the be-all-and-end-all of security solutions (almost nothing ever is regardless of what sales says).

Al Skierkiewicz

13-07-2012, 20:14

Everyone,
Now that the report is out, I urge you to read it in it's entirety, twice. Then sleep on it. Please do not cherry pick from specific parts of the report and draw conclusions about the system, the robot or the hardware in general. Note that each team is handled separately so that you can understand specific issues that occurred during the matches on Einstein. Please do not generalize a statement from a specific team report to indicate this occurs for all robots. (e.g. The Crio reboot time for Team 233 only was found to be 24 seconds.) Also, I urge everyone to stop using the generic term "communication failure" to describe the Einstein issues. The report is detailed enough that using that term is not descriptive of what actually occurred.
I would like to personally thank everyone involved in the Einstein weekend investigation. Each person was committed to finding answers, sharing data, and coming to a conclusion that would be of help to all of us. This will be one of those high points I refer to when asked why I continue to participate in this organization. There is a great group of special people involved. Special thanks, of course, need to go out to Bill Miller and Frank Merrick and all of the First staff who worked so hard since St. Louis to insure we continue to have a quality competition. Thanks to Jon Dudas and the FIRST Board for supporting this investigation and their continued commitment to excellence.

CalTran

13-07-2012, 20:20

If one of the winning teams had to do with this interference, or even any of the teams who played in Einstien, I would have agreed (of cource, not giving the interfering team any title). Since Jon Dudas's letter clearly states none of these teams were involved, I think FIRST made the right decision.

the point im getting at is, putting forth the person and team publicly would do far greater harm to the team and their image amongst their peers and the FIRST community. I think a much better solution to what Karthik wants, and is justified in wanting, is to have all the parties involved meet so that everyone can be satisfied as to reasons why, punishment, etc... But I do beleive that this shouldnt be known to the masses.

We are shocked, dismayed and troubled that an individual on a FIRST team would actually perform an intentional, malicious, wireless attack on our alliance. We are concerned that neither the individual nor the team he is associated have yet to come forward and publicly apologize for this horrendous incident. We hope that they come forward publicly soon, so we can all put this terrible event behind us. It would be a shame if they hid under the cloak of anonymity. Even if the team was completely unaware of the individual's actions, we would still hope that they would come forward, so that some of the motives would become more clear.

Words can't express how much this news hurts. To know that someone felt the need to intentionally target us for this type of attack stings beyond all belief. This is not the FIRST we grew up in, this is not the FIRST we love.

I believe Mr. Dudas' letter, as well as other posts, are attempting to make it clear that the actions of the individual should in no way place a stigma against the team that the individual was formerly associated with.

Bryan Herbst

13-07-2012, 20:21

After thoroughly reviewing the report, I have come to two conclusions:

1) FIRST did an incredible job of researching all of the factors that went into what happened on Einstein.

2) Any team, regardless of experience and ranking can have those simple little problems that could contribute to larger problems such as what happened on Einstein.

Regardless of whether or not the code and electrical problems were the root cause of the Einstein failures, each and every team should review this document and ask themselves whether or not they might have similar electrical/programming problems.

Loops without sleeps and bad crimps are problems that are easy to look for in your robot, but not necessarily easy to diagnose. If teams add these to their list of things to check on their robots during the build season, we can help ensure that robots are running to the best of their ability. Should any further malicious behavior take place, removing these problems as a potential source will assist in troubleshooting as well.

Ether

13-07-2012, 20:23

PayneTrain

13-07-2012, 20:26

While I still read, I think I should note to anyone who has hard feelings against 1114 or another successful team: what the heck are you gaining by beating someone else down? What would inspire you to go out of your way to see someone fail (the real kicker being you get nothing in return save for some personal satisfaction)? A bully beating up a bunch of kids in the FIRST playground is frankly depressing. I don't visit FIRST and FRC affiliated web sites to become depressed.

To make things a bit more light-hearted, I hope this act doesn't delay any development of SimPhone on Android. My Galaxy Nexus is a tame soul. It's letting me thumb through the report and push the link to some students on other teams.

Ether

13-07-2012, 20:38

I believe Mr. Dudas' letter, as well as other posts, are attempting to make it clear that the actions of the individual should in no way place a stigma against the team that the individual was formerly associated with.

^^Quoted for truth.

Furthermore, we should set the bar pretty high for characterizing an individual's intent to be "malicious".

The words in the report and Jon Dudas' letter are insufficient to support that conclusion.

We don't know the individual's motive or intent.

techhelpbb

13-07-2012, 20:44

^^Quoted for truth.

Furthermore, we should set the bar pretty high for characterizing an individual's intent to be "malicious".

The words in the report and Jon Dudas' letter are insufficient to support that conclusion.

We don't know the individual's motive or intent.

I agree completely. I can contemplate a few ways that someone might have been trying to actually help but picked a very bad way to achieve it.

Also, since there were issues outside of Einstein it sort of makes sense that there could be other ways this might happen that aren't quite so badly intentioned.

ttldomination

13-07-2012, 20:46

Oh man, I have never seen someone roasted so hard on CD.

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

In my short few years being around people, I've met a handful of people who are utterly brilliant but they have no social awareness and a lack of ability to see consequences. These people are nailed as socially awkward, but in their mind whatever they are doing, however they are doing it, is perfectly right.

Did this person interfere intentionally? Yes.
Do I believe that this person had malicious intent at heart? No.
Do I believe that the person has been adequately punished? Yes.

Additionally, I cannot even begin to imagine how team 1114 and their alliance partners currently feel, however, I believe that making a public apology is not the right path. The internet is the internet, and FIRST is FIRST. By publicly revealing himself/herself and the team, they leave themselves open to emotional, and possibly physical, harassment.

I am not condoning these actions, I am not condoning this person. However, without knowing all of the facts, I cannot support any action that would possibly bring this person any more harm.

- Sunny G.

dodar

13-07-2012, 20:56

techhelpbb

13-07-2012, 20:58

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

For all we know the person who they know had knowledge of it was just the tip of the iceberg. They could have told someone else that thought it was funny and did it to. They could have found this out from someone else that was also doing it.

There could also have been some other factor at work and let's hope it's one of those acknowledged within the report because a large amount of cost and effort went into this after the fact.

Racer26

13-07-2012, 20:59

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

This reminds me of a story told to me by one of my university profs in networking.

Some friend of his found a flaw in the City of Toronto's (IIRC, big metropolitan centre anyway) traffic light control system. Tried to tell city council and they ignored him, blew him off. So he turned every traffic light red in the city for several minutes. When they came after him? He said "Be thankful I didn't turn them green."

Brandon_L

13-07-2012, 21:01

Hmmm. Intentional interference... sounds like FRC is really starting to look like a "varsity" sport.

Aside from Einstein having issues there were tons of radio and cRIO issues at regionals. It really is time for FIRST to look at a new system.

After reading the report, I think I could say that I now support FIRST's claims of "Its not us, its you" in regards to comms loss at the regionals this year. If 5 of the 12 Einstein teams could have electrical/programming issues, I can see it being a larger problem then I thought before. Even though most of the errors were silly mistakes that wouldn't effect comss, imagine all the other teams that could make silly mistakes that would. The control system is fine, if put together correctly. More documentation is all you could really do about that, but teams have to read it. Now the radio, on the other hand..

I'm impressed with the detailed report and the handling of the situation by FIRST. Can't wait to get my hands on the FMS white paper!

As to the "interferer", I see his actions "accidentally intentional". From my understanding of the report, he simply tried to connect to the robot and provided an incorrect network password. I've seen robot networks on my mobile device many times, and tried connecting to it in the shop out of curiosity. Nothing really happened in that environment, but I'd give him the benefit of the doubt and call it a case of curiosity killed the cat. After all he did come forward.

LeelandS

13-07-2012, 21:01

I'm going to keep my personal feelings on the shelf for a while. I need to give it some time before I address more... unsavory... aspects of the situation.

But, what I am overjoyed to say, is FIRST did an amazing job at covering the situation. When I opened the document, I expected a 4/5-ish page summary report of what FIRST had been doing with the Einstein teams the past few weeks. I was pleasantly surprised to find an extremely long, fully detailed report of EVERY test and analysis run by FIRST. Bravo, FIRST. Bravo. You owe us nothing. Yet you went through everything for us. You guys rock!

I'll leave it with this. Do we really need a pound of flesh?
Just sayin'.

techhelpbb

13-07-2012, 21:03

I agree with you, but then also having these findings say that some of these teams lost connection because they were DDOS'ed and thats why they couldnt participate in matches, makes me say that the students, mentors, and parents of those teams deserve more than what this student has supposedly gone through. 1114 and 2056 specifically, because they were the 2 I know that said they were DDOS'ed, should feel the worst because their teams worked hard all season to accomplish what they did and got to the level they did this season and then had it swiped away by someone that had nothing to do with the match.

Every year someone that worked very hard has ultimate victory just out of grasp as there is a certain amount of random risk in this competition. I don't think that these teams are going away entirely empty handed. Though I do respect that they got a raw deal.

We can't assume that the person involved is any more guilty than a hypothetical weird wiring issue on the field.

We don't know if they were the only one doing this at all.

We don't know if they fully understood what they were doing.

We don't know how it was if they were trying to fix it they were dismissed.

I can't feel the need to ruin what could be some person's life when these teams will continue and have more chances to make it that far.

Besides this is perhaps one of those stranger and more memorable moments where the memory of the participation is more valuable than the dusty trophy. I'm sure a great number of us won't soon forget what happened.

Alexa Stott

13-07-2012, 21:17

Imagine the chaos if FIRST allowed any random person to walk in and give their suggestions while trying to diagnose the issue during Einstein. They'd be flooded with every person out there who has a theory (take a look at the numerous Einstein threads filled with people bickering about what they think happened to get an idea of how many people that would be) to step in.

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match. If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to? Because of those facts, I think it's hard to say that the intent was not malicious.

My heart goes out to The Eh Team. Nobody doubts that they were an amazing alliance of 2 of the best teams (and one very promising up and comer). When people look back on Rebound Rumble, they will not only remember that 16, 25, and 180 won, but that 1114, 2056, 4334, 987, 233, 118, 548, 2194, and 207 were all outstanding teams whose robots excelled at the game.

Where does his letter clearly state that [none of the teams on Einstein were involved]?

In Jon Dudas' statement, he says:

The person who engaged in this interference was associated with an FRC team, but not with any team on the winning alliance.
Nothing about the other Einstein teams, though.

Jedward45

13-07-2012, 21:21

Im really really glad FIRST took their time to put together an awesome report! BUT.... after reading the whole things twice, I'll admit I don't quite have the technical experience to understand the report in its entirety.

Could someone summarize and explain the more detailed aspects of the Robot Testing and Failed Client Authentication Testing? Specifically, what intentional interference actually happened, how did it cause problems, and what are they planning to do to fix the issue?

Thanks!

Chris is me

13-07-2012, 21:21

Firstly, thanks to everyone in and out of FIRST who made this exhaustive testing and report possible. It is great to have such a thorough analysis of the forces at play on Einstein and the lengths they went to replicate on field conditions were extraordinary. I sincerely hope this leads to dramatically less communications faults at any event this year.

It's absolutely appalling that someone in FIRST would sabotage an alliance (and then some) by exploiting a security vulnerability. It's also appalling that it could be so simple to knock a robot out of commission in any FRC match since Week 4. Cisco's got some 'splainin' to do...

Did this person interfere intentionally? Yes.
Do I believe that this person had malicious intent at heart? No.
Do I believe that the person has been adequately punished? Yes.

I'm having trouble processing the idea that someone could deliberately attack all of the robots on an alliance by exploiting a security vulnerability without "malicious intent". Perhaps they were trying to raise attention of an issue FIRST should have known about (just about the only scenario I can think of that would even resemble "good intent"). Einstein is by no means whatsoever the proper time and place to demonstrate this problem.

And does it even matter what his / her intent was? Are the affected teams supposed to feel better about being cheated out of a fair chance at victory because "oh, he / she had good intentions"?

techhelpbb

13-07-2012, 21:23

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match. If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to? Because of those facts, I think it's hard to say that the intent was not malicious.

As the report points out a not trivial amount of problems were actually on the robot. This would make it hard to see cause and effect while testing such an attack. You wouldn't know if the attack stopped the robot or the robot stopped for reasons you had nothing to do with. Just like the FTA and other FIRST folks wouldn't know if the robots were broken or someone was causing trouble.

Add to this the storm and the other distractions it's just a perfectly bad combination.

PayneTrain

13-07-2012, 21:24

I know I need to let this all marinate, but if the cause is so noble there is a far better way to achieve the results without taking it out on an alliance that has quickly become a punching bag in darker corners of the community (even though a member of it is the newest team in the FRC HoF...)

:/ this is rough stuff

connor.worley

13-07-2012, 21:25

This reminds me of a story told to me by one of my university profs in networking.

Some friend of his found a flaw in the City of Toronto's (IIRC, big metropolitan centre anyway) traffic light control system. Tried to tell city council and they ignored him, blew him off. So he turned every traffic light red in the city for several minutes. When they came after him? He said "Be thankful I didn't turn them green."

I think taking down Einstein is the equivalent of "turning the lights green" here though.

Brandon_L

13-07-2012, 21:28

It's in Jon Dudas' statement (third paragraph):

The poster he is referring to stated that non of the Einstein teams were involved, not none of the winning alliance.

Alexa Stott

13-07-2012, 21:32

As the report points out a not trivial amount of problems were actually on the robot. This would make it hard to see cause and effect while testing such an attack. You wouldn't know if the attack stopped the robot or the robot stopped for reasons you had nothing to do with. Just like the FTA and other FIRST folks wouldn't know if the robots were broken or someone was causing trouble.

Add to this the storm and the other distractions it's just a perfectly bad combination.

S/he was asked to stop and did not comply. The person also attempted to inform officials about it, indicating (to me, at least) that s/he was aware that what they were doing could have been causing issues.

If it had been one match that this happened in and had the person put the phone away when asked, my feelings on this would be much different.

techhelpbb

13-07-2012, 21:42

was[/i] asked to stop and did not comply. The person also attempted to inform officials about it, indicating (to me, at least) that s/he was aware that what they were doing could have been causing issues.

If it had been one match that this happened in and had the person put the phone away when asked, my feelings on this would be much different.

Then again putting the phone away doesn't mean much if you think about it. I can leave an application running with my phone screen off on any Android device.

So they put it in their pocket and it's still causing trouble. Then they take it out and it's still causing trouble. They are likely just as distracted as everyone else with the weather, the people watching, people's behavior near them, etc. It's really hard to say what people do when they put themselves in a bad spot like that. From the second they were asked to put it away they were in fact in a lot of trouble.

I don't disagree with your gut reaction. After all I had concerns about power issues before this all started and FIRST suggested we not test at Einstein those issues. As it turned out some of those issues existed who knows maybe they would have been found (I can't prove it either way). I even went so far as to ask the question in the official forum with regards to the championship.

However, no means no. We completed the testing of my little oscilloscopes for the power to the radio on off season events. We did it where it would do as little harm as possible to FIRST proper if something happened that was not expected. We did it with full knowledge of those who could be effected.

There was most definitely bad judgement at work with this person's choice of actions. They've made themselves a sitting duck for suspicion. It's highly unlikely we will ever know if they are the only source of the deauth issue.

The amateur way they handled themselves however, concerns me because usually where there's an amateur trouble maker that is easy to spot it is a distraction for someone that's not an amateur.

I mean not to start a witch hunt just to make the point. We live in a complicated world where rarely do specific effective security issues just magically appear to a single person. Like the systems themselves they are collaborative efforts. I have a hard time believing that a person so clearly asking to get caught just figured this all out on their own.

In a perverse way the drive to seek out the problem with the WiFi might have guided them to this issue via collaboration, but even then I doubt they crafted the attack and were ready to report it just in time of that one event.

I do computer security for a living in part so perhaps it's just in my nature to rarely assume the singular genius that hands themselves to me on the silver platter isn't covering for someone else.

Ether

13-07-2012, 21:49

Imagine the chaos if FIRST allowed any random person to walk in and give their suggestions while trying to diagnose the issue during Einstein. They'd be flooded with every person out there who has a theory (take a look at the numerous Einstein threads filled with people bickering about what they think happened to get an idea of how many people that would be) to step in.

Alexa, I think you are misinterpreting the previous posts. No one is suggesting that any random person should be allowed to walk in and give their suggestions while the FTAs are trying to diagnose a field issue.

Also, the document indicates that this individual was observed doing this multiple times and continuing throughout the match.

The document says that there was a witness(es) whose description of a person using a phone matched the description of the person on the field earlier. Perhaps it was the same person.

If they were just doing some unauthorized (and incredibly harmful) troubleshooting, why didn't they stop after forcing a robot to lose connection the first time? Why did they take their phone out after explicitly being told not to?

We don't know.

Alexa Stott

13-07-2012, 21:56

The document says that there was a witness(es) whose description of a person using a phone matched the description of the person on the field earlier. Perhaps it was the same person.

FIRST officials spotted him twice:
This individual was asked to put away the cell phone, and complied. Later, the individual was observed using the cell phone again, and at that point, before the last two Einstein matches were played, was asked to leave the field area, and did so.

Hjelstrom

13-07-2012, 21:57

Joe Johnson

13-07-2012, 21:58

My head spins there is so much to take in.

First, I join my voice to the chorus that is saddened and outraged by the behavior of this individual*. I don't suppose that this is the first malicious attack by an individual on another FIRST team or alliance but this attack is so brazen. I am sick. Really sick.

Second, I don't think that it is too much to expect the former team of this individual to apologize to the teams involved as well as the entire FIRST community. What is more, depending on the circumstances, I would think that it may be appropriate for them to assure the FIRST community that they are committed to Gracious Professionalism in its highest sense and that they are committed to rooting out anything within their team culture that may have contributed to fostering the behavior of this individual.

I don't want to start a rumor but does anyone else think that this must be related to the nonsense that went on at the Greater Toronto East Regional? If so, the Canadian FIRST community really has to work to lance this boil.

Third, I am really disturbed by this statement in the report:
FMS White Paper – FRC will be producing a white paper which describes how the Field Management System operates. This will include details on the topology of the system, components used and the communications paths between the various components.This seems to imply that they don't have this now. Wow.

Fourth, they offer "Additional emphasis in training and documentation" as a mitigation for "Robot D-Link radio reboot due to power dip." I hate this proposal. It seems to me that either the robots must return to active duty (passing packets back and forth) in a handful of milliseconds OR the radios must work down to a voltage that is almost literally impossible to get to without tripping the breaker (say 1.5 volts or something ridiculous like that).

Fifth, it seems to me that FIRST (and the FMS) has one implied contract with the teams: We will get X% of your data packets from your Operator Interface to your Robot and vice versa within Y msec. In my view of the world, literally the SECOND the FMS breaks this contract for even one robot on the field, the match has to stop and be replayed. Period.

With the current system this vision cannot be fully implemented because of dead/disconnected batteries and other complications.

I will say that if FIRST had this as their standard, this attack would not have been prevented but the attacker's purpose would not have been accomplished, which may have kept the attacker from even trying.

Reading the report, it is hard to know if the FMS system actively monitors this contract... ...past is past. Going forward, I really think FIRST should implement such a system hosting another tournament.

Finally, I hope that the FIRST community can pull together as a result of these events. We must do our best make FIRST better because of it. I have faith in our ability to do so.

Regards,
Joe J.

*I say "individual" because I am not sure if there were two or one. The report seems to indicate two folks were involved but only one was banned for life so... ...perhaps I am misunderstanding the text.

Andrew Lawrence

13-07-2012, 21:59

kazzykaty

13-07-2012, 22:03

I think that FIRST did a great job at handling this situation. This is something that could have easily been ignored or swept under the rug. No one would have known. Instead, they put a lot of time and money into the investigation.

It is a shame that something like this would occur in a student-based organization but it is comforting to know that FIRST puts its participants and ethics first.

I am so proud to have been a part of such a great organization.

Tom Line

13-07-2012, 22:04

The larger issue than who did this is how was the system allowed to get to the point that it was possible at all. Let's worry about all the other D.O.S. (denial of service) attacks we have yet to find.

Clearly changes need to be made. It took extraordinary effort on the part of too many people to resolve even these issues to this point.

It still goes back to the assumption that the system is above flaw and that assumption being incorrect. In this case the system has a security issue and an active exploiter. Take if from me: you can look for and fix security issues before they get exploited as best you can or you can wait until they cost you reputation, resources and opportunity. Had they even profiled the issue before hand they could have dramatically reduced the chaos after the fact (if you don't fix it at least acknowledge it exists at the remediation level).

I disagree entirely. I don't believe anyone believes (or believed prior to Einstein) that the system is above flaw.

Take any system, no matter how well designed, and subject it to 60,000 ambitious folks all playing with it and see how secure it is.

This week's 'Yahoo' password hack displays just what happens when even the most competent network security is open for public interaction. Someone WILL find a way in. Google, Microsoft, and even the stock market have been subject to security invasions as well.

I hate to say it, but in this situation security through obscurity is FIRST's best bet. The entire system needs to be removed from the consumer electronics spectrum that all these common tools are designed to work with. I.e. - standard a/b/g/n wireless needs to disappear. If this does not change and go to a proprietary system, I will 100% guarantee you WILL see this happen again.

akoscielski3

13-07-2012, 22:04

Billfred

13-07-2012, 22:05

Oh man, I have never seen someone roasted so hard on CD.

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

In my short few years being around people, I've met a handful of people who are utterly brilliant but they have no social awareness and a lack of ability to see consequences. These people are nailed as socially awkward, but in their mind whatever they are doing, however they are doing it, is perfectly right.

As much as I can, I use the concept of gracious professionalism as an internal yardstick. However, I have a really hard time believing anyone would think that the correct way of carrying yourself when in possession of an issue like this is to disrupt the climax of FIRST's largest, most-anticipated, most-covered event.

I believe there are far better ways to demonstrate these bugs. FIRST has held Beta Day in Manchester. If the FTA and some volunteering teams were on board, I'd be okay with someone demonstrating a novel flaw the night before SCRIW and forwarding the information to FIRST. And if you feel the urge to break something at Championship, why not practice matches with your own team?

The phrasing of Jon's email makes me believe the individual involved is (well, was) a mentor on a team. As the guy in denim says, we get the best of what we celebrate. There is no room for celebrating interference with any competitive FRC match, whether it's Einstein or Q12 at some Box-On-Wheels Extravaganza of a regional. There is plenty of room to celebrate mentors that discover field issues and disclose them responsibly.

Ether

13-07-2012, 22:09

http://www.chiefdelphi.com/forums/showpost.php?p=1177303&postcount=115

Ether

13-07-2012, 22:14

There is no room for celebrating interference with any competitive FRC match, whether it's Einstein or Q12 at some Box-On-Wheels Extravaganza of a regional.

Just to be clear, you're not implying that anyone who has posted to this thread is suggesting that "celebration" is the appropriate response to what happened, correct?

Ether

13-07-2012, 22:19

And does it even matter what his / her intent was?

Yes. Intent matters.

LeelandS

13-07-2012, 22:19

I actually would prefer this team remain anonymous. Reason is that the Team itself is not responsible (or so it seems). And what ever the mentor/student/parent did wont ruin the whole teams reputation. If the team is revealed the team will become ashamed for what another individual did and could possibly shut down. I would hate to see this happen. Its not the Team's fault and I would hate to see them suffer from it.

Just my $0.02 CAN

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people :) But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

Alan Anderson

13-07-2012, 22:21

Could someone summarize and explain the more detailed aspects of the Robot Testing and Failed Client Authentication Testing? Specifically, what intentional interference actually happened, how did it cause problems, and what are they planning to do to fix the issue?!

There's a bug in the version of firmware used on the field access point in Week 4 and later. It manifests itself when a D-Link with Rev A hardware is connected to the access point. If a "rogue" client attempts to connect to the wireless network and provides an incorrect WPA security key, the wireless link between the robot and the access point is disrupted. A subsequent failed connection attempt by the rogue can restore the robot link.

The "intentional interference" was someone using a smart phone to try to connect to the network established for one of the teams playing a match and failing. That should have resulted in exactly zero effect on the networks, but the firmware bug caused that team's robot to lose its link to the field network.

As an immediate fix, they're reverting to a previous version of firmware for the access point. Testing showed that version not to be vulnerable to the Failed Client Authentication problem. The ultimate fix will involve fixing the bug, including a test for the problem when doing validation acceptance for new revisions of firmware, and putting specific features in place to detect and log such attempted connections.

dodar

13-07-2012, 22:23

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people :) But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.
I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

If you did do that, then we wouldnt be having this conversation, which would in turn make you not go and invent a time machine. So seeing as that didnt happen, either you fail in doing so or you do invent a time machine but you get taken out by the duplicate paradox...of course if that happens, which I hope it doesnt, you in essence die the day you go back in time to stop yourself from screwing up at last year's championships. :p

ttldomination

13-07-2012, 22:25

And does it even matter what his / her intent was? Are the affected teams supposed to feel better about being cheated out of a fair chance at victory because "oh, he / she had good intentions"?

In the grand scheme of things, it doesn't matter. The person was punished, and it appears that it's an acceptable method.

However, intention is something that turns a situation on its head. Intention is what's driving every single post about how appalled people are that this happened. We are assuming intent, and therefore we are writing posts filled with hate.

I will iterate that I cannot begin to imagine how the effected teams are feeling, and the goal of my post was not to curb their anger/disappointment, to that, I have no right. My goal was to simply promote an ounce of open minded-ness.

- Sunny G.

apalrd

13-07-2012, 22:26

Since we're (mostly) engineers on CD, let's think of this as an engineering problem. There is a supplier-customer relationship where FIRST is the supplier (of the control system) and the teams are the customers (of the control system).

The customer must have requirements for the supplier. The basic requirements for an FRC control system by a team are:

-Must be easy to setup for a very inexperienced team. Even if the manuals are clear, complex systems are inherently easy to misconfigure even if the directions specify how to do it.

-Must have a boot time of no more than (x) seconds for the entire system. I would be willing to exempt the DS only because I know how slow Windows is. This time limit is to the advantage of teams who want to run quickly, and the field crew which can recover from errors more quickly.

-Must have a certain amount of minimum functionality that can be achieved with minimal computer skills by a team - I would include a requirement for a certain amount of default code, at least mapping JS axis/buttons to motors and solenoids, since I know how many teams previously relied on default code. For some reason, the current control system lacks this and I don't really like it.

-Must be protected from any interference - I believe this requirement previously existed but was not fully met. I will discuss this later.

-Must have certain safety checks implemented - Specifically loss of communication with driver station, crashing of team code, and network error. Currently the first two are not implemented well, the third is implemented with a packet CRC which is good.

Good requirements? I think this covers the basics.

Now how do we verify that we've set all of the requirements correctly? How do we set the exact parameters of the requirements (boot time, amount of sophistication of hacking attempts, default code, watchdog timers, etc.)? Area they created by the supplier arbitrarily, or does the supplier actually know what the proper values are from their experience in designing control systems? Does the supplier actually work with control systems like these, or are they guessing based on experience in another field (say, industrial controls vs automotive controls - Timing and power requirements are *very* different between these two).

There are a few key flaws in this system which are highlighted above. Who is making the requirements that define boot time, maximum time without communication before the robot disables itself, or the minimum sophistication of a hacker attempting to compromise the security system? Do the requirements even exist, or is it more of a best-attempt system for some of these variables?

Best-attempt parameters always ends up as a variable which can be compromised to meat any other goal, with no real loss to the supplier.

The next step in engineering is actually designing the system. So, let's skip that step and end up with what we have now.

Once the system is engineered, you have to verify that you have met all of your requirements. How do you test this? While nobody thought to test the case of 802.11 access requests stated in the document, and this is OK given the obscurity of the bug and it's existence in hardware from another supplier, I can guarantee that somebody thought about DOS attacks on 802.11 (if they didn't, they would have not come close to meeting the requirement set above). What is the solution to them? The AirTight device was clearly chosen to detect DOS attacks, without adequate testing to verify that it actually met the requirements.

I am not pointing blame to anyone or any company specific, as I know there are many people and companies involved in the design of the system, but it has issues on a whole that are nobody's fault.

This document clearly shows me why 802.11 is not used in critical applications in this way (open SSID broadcast especially). No offense to FIRST, but the choice for 802.11 is probably the largest design failure of the entire control system (including my rants on compile/download/boot times and stuff). This testing, specifically the individual who is anonymous, shows just how vulnerable the system is, and how little it is protected from any intention or unintentional interference from a device that everyone carries in their pocket.

As a final thought, if the issues are blamed on the teams in such large quantities, then something must be too complicated, ambiguous, or otherwise error-prone in the system (basically the system is too complicated if that many people can't get it setup correctly).

Someone earlier in the thread questioned the failure at GTR as possibly being a related event. There were more failures than just GTR (although it was the most publicized) that showed up in exactly the same way as what happened to Simbotics. Teams' radios were blamed by FTA's, and everybody lived unhappily because the radios did suck (they thought) and there was nothing they could do to change it. The FMS was "above blame" - Because the DS could not ping the teams radio but the other 5 DS's could ping the other 5 radios on the field, it must have been the fault of the team's radio. (on a related note, the FMS gathers very little logging data for itself, most of it is collected by the DS and forwarded to the FMS). The general idea is that, since the cause isn't definitely the field, it's probably the teams (especially since so many teams have so many issues with the radio power wiring) and then it becomes always the teams fault, and the FMS is never blamed for failure of communication.

Anyone want to suggest another air interface? I've been thinking about a few...

Chris Fultz

13-07-2012, 22:29

This (similar) post is in the thread with the FIRST letter and link, but i was requested to duplicate it here -

There will likely be several threads and posts about the report and the contents. Please remember this is a public forum, use caution and care in what you say and what you claim.

Also, please read the report before commenting on it. There is a lot of information, and many questions and comments may be answered in the report itself.

Regarding the report, this is a detailed summary of the fact-finding, the process, the testing and the results and conclusions found by a large, diverse team. You may or may not agree with all of the conclusions drawn, but there is a great deal we can all learn from the detail of the report.

We also owe a lot to the 12 teams involved and their level of participation at the FIRST weekend and data collection process.

Also, note the request from FIRST for input. If you have constructive input, please use the email address to provide that to FIRST.

* Addition -

Read the report and the detail for what it actually says, not for what you think it says. And, to keep the thinking clean, if someone make a conclusion or statement not supported by the report, then call them on it and clear it up. Some are doing this already and it keeps the conversations and conclusions accurate.

cgmv123

13-07-2012, 22:32

Gigakaiser

13-07-2012, 22:33

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people :) But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

I disagree - the team (or whoever on the team is aware of the individual) should release the information and acknowledge that it was an individual, not a team effort. Imagine the reputation of the team if the information came out sometime in the future by another source - it would not look good. If the team came out it would be seen as a gracious step forward. I highly doubt the entire team was aware of the individual during einstein anyway.

Alexa Stott

13-07-2012, 22:35

or her. The report does not identify the gender of the individual.

The male pronoun is often used in cases of ambiguity. That you took the time to take issue with this incredibly trivial part of my post indicates that maybe we all need to step away from our keyboards for a bit.

dodar

13-07-2012, 22:37

I disagree - the team (or whoever on the team is aware of the individual) should release the information and acknowledge that it was an individual, not a team effort. Imagine the reputation of the team if the information came out sometime in the future by another source - it would not look good. If the team came out it would be seen as a gracious step forward. I highly doubt the entire team was aware of the individual during einstein anyway.

Wouldnt matter. From then on that person would not be known for their specific name but as a member/former member of team _______.

Gregor

13-07-2012, 22:38

I don't want to start a rumor but does anyone else think that this must be related to the nonsense that went on at the Greater Toronto East Regional? If so, the Canadian FIRST community really has to work to lance this boil.

If you honestly believe that you can draw parallels between not attempting the coop bridge with very very good teams and deliberately sabotaging the premier FRC matches, I don't even know what to say... You can tell that there is some animosity towards these outstanding teams if you went to a Canadian regional, but if you think that someone wants to ruin their Einstein appearance for some twisted payback, I think you're sadly mistaken.

Please don't make generic assumptions when you have absolutely no proof, only speculation on whom it may have been. You know what they say about assumptions...
---

I believe that FIRST staff, volunteers, and Einstein teams handled the situation exceptionally well given the circumstances. Lets not forget that 12 incredible teams went onto the Einstein field, and 12 incredible teams left it.

For this one person who deliberately tried to sabotage the event, there are literally hundreds of thousands of people who would condone in. Lets remember that one rogue person shouldn't be regarded as a significant change in FIRST culture.

shawnz

13-07-2012, 22:41

The issue isn't just trying to connect to the network, the issue is trying to connect to the network - while entering an incorrect WPA key

Keep in mind that no key == an incorrect key, although I don't know if that applies in this specific attack. If it does, though, it increases the likeliness of it happening accidentally greatly.

Also note that it seems only some versions of the AP firmware have the bug, so this is not a long-standing issue.

Fifth, it seems to me that FIRST (and the FMS) has one implied contract with the teams: We will get X% of your data packets from your Operator Interface to your Robot and vice versa within Y msec.

AFAIK, this is basically what QoS is for, and the way they've described it in the report, it seems they're not using it yet (but will be next year).

As a side note, I'm also in the "stupidity rather than malice" camp, but from the looks of it, it wasn't just one person responsible. Either way, the actions taken were appropriate IMO -- including the provided anonymity.

Gigakaiser

13-07-2012, 22:43

Wouldnt matter. From then on that person would not be known for their specific name but as a member/former member of team _______.

Should the team then remain anonymous to avoid potential scrutiny? Again, what if the information was released by another source? Coming forward now would be the best way to avoid scrutiny at a greater scale.

Greg McKaskle

13-07-2012, 22:43

Could someone summarize and explain the more detailed aspects of the Robot Testing and Failed Client Authentication Testing? Specifically, what intentional interference actually happened, how did it cause problems, and what are they planning to do to fix the issue?

To add to what Al suggested, if you don't understand something in the report, this forum is a great place to ask for further explanation. The above is a good example. It is nearly impossible for all items in the report to be understandable to all that read it.

Alan did a great job of answering the question, but I'll add that the FCA is not itself anything to be worried about. It could be due to a typo, a forgetful user, or an attempt at cracking the password. The access point with logging enabled may log the mac address of the FCA client.

As mentioned in a previous post, the FW bug means that teams who used encryption for build season, opened their programming laptop at the event and unknowingly attempted to connect to their own robot with their build-season password resulting in an FCA. This was investigated to determine if it contributed directly to issues on Einstein.

Greg McKaskle

Greg McKaskle

Racer26

13-07-2012, 22:44

Also, the biggest crime against 1114 at GTR East was the ramming of the already balanced (1219/1114) coop bridge by 2185.

2185 was not at CMP.

dodar

13-07-2012, 22:46

Should the team then remain anonymous to avoid potential scrutiny? Again, what if the information was released by another source? Coming forward now would be the best way to avoid scrutiny at a greater scale.

If FIRST isnt the one to release that information after getting right from the team, if this person is even related to a FIRST team, then the source should be punished as much or even harsher than the original culprit. For someone else other than FIRST, the culprit, or the team(if there is one) they are only trying to stir up emotions by releasing that information.

BrendanB

13-07-2012, 22:46

Thank you FIRST for your transparency and releasing this extremely detailed documentation to us in the FIRST community. A huge thank you to those who worked extremely hard on creating this report and to those involved in the thorough investigation and testing!

With that being said, what is done is done. FIRST has dealt with said individual, and everyone is attacking him like wolves a sheep. While you do have reason to be angry (especially those who competed on Einstein) what is this solving? We can't determine one's intent and if I knew who was involved I would hold such a huge prejudice towards that team/individual without reason. I'm angry that this happened but at the same time robot hacking was bound to happen eventually and it has crossed my mind over the years (not with malicious intent) but I wondered when the day was when we would see it play out. It is a shame when, where, and to whom it was done but you can't change it. I'm glad FIRST is looking into solutions to fix these problems to eliminate threats in the future.

One thing that stuck out to me was the detailed explanations of electrical/programming issues experienced during the testing. I have a new respect for our electrical/programming teams as even the smallest issue of a loose wire can cause so many problems.

my $0.02

aspiece

13-07-2012, 22:47

Truck Town Thunder, FIRST Team 68, would like to officially support FIRST in the results as well as applaud them for the way it was handled. Situations like this are unfortunate and it can be difficult determining the best solution to this type of problem. FIRST Team 68 supports FIRST in their decision.

akoscielski3

13-07-2012, 22:51

If you honestly believe that you can draw parallels between not doing the coop bridge with very very good teams and deliberately sabotaging the premier FRC matches, I don't even know what to say... You can tell that there is some animosity towards these outstanding teams if you went to a Canadian regional, but if you think that someone wants to ruin their Einstein appearance for some twisted payback, I think you're sadly mistaken.

Please don't make generic assumptions when you have absolutely no proof, only speculation on whom it may have been. You know what they say about assumptions...
---

I believe that FIRST staff, volunteers, and Einstein teams handled the situation exceptionally well given the circumstances. Lets not forget that 12 incredible teams went onto the Einstein field, and 12 incredible teams left it.

For this one person who deliberately tried to sabotage the event, there are literally hundreds of thousands of people who would condone in. Lets remember that one rogue person shouldn't be regarded as a significant change in FIRST culture.

I agree with Gregor.

Also, Please dont start abolishing FIRST in Canada over a few things. I personally Love 1114, and 2056. And I KNOW that a lot of people from other Canadian teams like them too. They should be looked up to. I talk to student from both their teams and they are extremely nice and always up to talking. Dont start saying there is a problem with FIRST in Canada, because you don't know what you're talking about. (This response isn't to Gregor, but the person he quoted in his original post)

THE END.

Andrew Lawrence

13-07-2012, 22:55

Should the team then remain anonymous to avoid potential scrutiny? Again, what if the information was released by another source? Coming forward now would be the best way to avoid scrutiny at a greater scale.

I think the best solution (to make everyone happy and minimize damages) would be for the team to come out and say it was someone from their team who acted unintelligently at Einstein, they apologize for everything, Gracious Professionalism is their greatest priority, they hope the FIRST community will find it in their hearts to forgive them, etc etc, The individual's actions do not represent the team, and most importantly the individual will not be named.

With that, the people "get" the culprit, the team apologizes and gets a cleared name (and even some respect from us for doing the right thing), and nobody can crucify the person at wrong because they don't know who it is.

Although, to be completely honest, I don't see why needing to know this information matters. If the team doesn't want to come out in the open, it's completely understandable, and it doesn't make us as people any better knowing who did what.

Ether

13-07-2012, 22:55

The male pronoun is often used in cases of ambiguity. That you took the time to take issue with this incredibly trivial part of my post indicates that maybe we all need to step away from our keyboards for a bit.

It wasn't trivial, just too obscure.

It appeared you were assuming something* that wasn't stated explicitly in the report. Other assumptions not supported by the report are largely responsible for the widely differing opinions concerning the appropriateness/adequacy of the actions taken by FIRST.

*At the time of posting, I hadn't noticed that in a previous post you had used "s/he" and so you were obviously aware that the report did not specify gender. So I stand corrected.

Ekcrbe

13-07-2012, 23:01

I don't know about those two, but after those semis 68 had unexplained comms issues in the finals, and they do use labview

I'm not entirely sure about that incident, but I had a sneaking suspicion even before Worlds that we were somewhat vulnerable to dropouts when being hit near the battery. It could have been that, considering it coincided with a collision right there with the corner of another robot.

Plus, it was only once, and the NI experts in the pits actually reviewed our code and found it to be fully functional and efficient enough (~50% CPU usage average).

apalrd

13-07-2012, 23:07

I think you're forgetting advanced functions, speed and customization for advanced teams. It can't be so simple that advanced teams can't take their robots to the next level.

I'm glad you brought this up. While there is a need to support advanced functions, there is also a limit to how far you can go reasonably.

How many teams actually use the second digital sidecar? What about the second solenoid board? How many use a solenoid board at all? Could you replace the solenoid board with Spike relays and be just fine? If you were to remove support for the second sidecar entirely, how many teams would actually be affected, and would it be an issue which they could not possibly work around?

What about the software end? How many teams actually use the DMA functions in the FPGA? What about the vision processing - Do we really need to do this on the cRio? What percentage of teams found that to be a good solution this year?

Think of all of the teams that did it the "easy way", and used the default dashboard to see the camera image, then added tape to their computer screen to indicate where the target should be. They outnumber the teams that actually did the processing on the cRio, since it was so hard to do reliably without disrupting other systems.

In my programming experience, an experienced control systems programmer can do advanced things with very little framework, given a few key attributes exist:
-You can rely on a deterministic execution timer - It is acceptable to skip this rule if you have a reasonably accurate idea of the actual iteration time, and the execution time isn't too large (anything under 30ms should be fine for FRC applications).
-You can rely that the data given to you is valid OR you can validate this data in some way (if a missing/ejected cRio module can cause valid but old data to get to the user program, it is impossible to detect and thus a failure of this requirement)
-It is assumed that raw read/writes of the IO is available (e.g. must be able to sample all inputs at the execution rate, and set all outputs at the execution rate), as this will be used by inexperienced programmers as well.

Having worked in the "other competitive robotics league" of VRC and used both EasyC 4 and RobotC 3 (the Cortex versions, respectively), I was rarely faced with a limitation imposed by the (very limited) programming environments. In particular:
-RobotC did not have the timed task structure that EasyC had. I could take delta_t and use it to adjust the wait, and that was accurate enough (plus I could use delta_t to compensate for any execution time errors).
-EasyC failed to directly tell me the state of the field (e.g. Enabled/Disabled/Autonomous), as it had a funny way of calling and terminating functions. I found that this violated the third rule, and this was my primary cause for switching to RobotC.
-RobotC language did not allow pointers in any form. This was limiting as most of my math library relied on passing pointers to functions for operating, but I resolved it.

The cRio currently does this:
-In LabVIEW, an RT task meets the first requirement above while consuming an exorbitant amount of CPU resources in the process. This is fine, but the other inefficiencies of LabVIEW push the CPU load through the roof. A normal task fails the first requirement, but like RobotC, we know the timing so we can compensate (and it's generally acceptable, although the timing of LV is much more unstable in non-RT tasks than RobotC).
-The last time I checked, it fails the second requirement above if the module comes out of the cRio after boot, as the FPGA calls return the data that existed when the module was removed (in our case, it ejected itself from the cRio while crossing a bump). I do not know if this has been fixed lately, I haven't checked.
-It passes the third requirement.

This illusion of more functionality (and the larger processors to go along with it) often just adds to the complexity of the system on whole for marginal gains. While marginal gains are good, the marginal gains in performance seriously affect the usability to new teams, and, to me, that is worse than a very simple system which reliably performs the "easy task" and forces teams to do live within the limits of the system to perform the advance tasks. While I'm not saying the old IFI PBASIC system was good, the PIC based system solved most of the advanced functionality issues the PBASIC one had, and was reliable and it booted fast. An IFI system with a small ARM user processor like the Vex Cortex would be amazing

You should never compromise the base functionality in search of advanced features. Ever.

Barry Bonzack

13-07-2012, 23:17

Thank you to everyone involved for the thorough investigation, and communication to the rest of us.

The report said this

LIGHTING TEST
One of the differences between the Einstein field and all four of the division fields was the lighting conditions. The purpose of this test was to investigate whether that difference could cause control or connection issues. Lighting was brought in to replicate the

But I read this

LIGHTNING TEST
One of the differences between the Einstein field and all four of the division fields was the lightning conditions. The purpose of this test was to investigate whether that difference could cause control or connection issues. Lightning was brought in to replicate the...

...I have no other comments to make at this time.

LeelandS

13-07-2012, 23:17

The individual's actions do not represent the team, and most importantly the individual will not be named.

Unfortunately, while we on Chief Delphi like to maintain that one member of the team doesn't represent the entire team, it doesn't always work out like that, especially in cases of extreme circumstances. Now, you, or I, or anyone else on CD may believe the individual doesn't reflect the team, but there will always be people who believe otherwise. Any way you slice it (and I, personally, don't put much stock in this), people could look at it as "Well, this is the kind of people Team Such-and-Such has."

An individuals actions DOES, in fact, reflect the team they're associated with to some degree. Which is why I think it is best that the team and the person in question remain anonymous.

Dustin Shadbolt

13-07-2012, 23:30

I am seriously stunned. I'm stunned that someone would do this especially at an event like FRC. Sounds like a personal vendetta. However, I applaud that FIRST is being so transparent and that they securing the problems for the off-season events.

I think we all are upset and stunned, but we need to learn from it and move on. Yes this is detestable behavior from someone especially someone associated with FIRST. However, we can't let this hiccup cause us all to fight among each other and become sinister with each other. We learned a lesson that FIRST isn't magically protected against interference based on the good will of the participants. There are and will be more bad apples. It's the double-edged sword of working with kids/mentors that are gifted with technology. Hopefully the problems are fixed and we don't ever have to face this issue again.

-Dustin Shadbolt

Greg McKaskle

13-07-2012, 23:46

Third, I am really disturbed by this statement in the report:
Quote:
FMS White Paper – FRC will be producing a white paper which describes how the Field Management System operates. This will include details on the topology of the system, components used and the communications paths between the various components.
This seems to imply that they don't have this now. Wow.

In case you interpreted that section in this way, the white paper isn't needed to document the system -- there are dozens of engineering requirements and development documents, but to publicly release a comprehensible description of the system so that all issues involving robots or field can be resolved more quickly, with less guesswork.

Greg McKaskle

Ekcrbe

14-07-2012, 00:03

While this is a terrible event (it's terrible, it happened, end of story), my concern is that field issues happened all around the world during regionals/districts. Was a similar event the cause of all of those disruptions? There are people in the world who sometimes make the wrong decision, but to have such a person at all of the events where connection issues were present doesn't seem right.

As the report showed, there were certainly a lot of issues with communications, coming from a lot of sources. And those are just 12 of the best teams in the world. Team error likely supplied most of the connection troubles throughout the season, just as it was the root of many incidents on Einstein. I severely doubt this is a repeat offense from earlier in the season.

With that said, let's look at the big picture. Every coin has two sides, even this one. It's not even close to fair what this "individual" did to those teams, or even their opponents. It never will be. But trying to look backwards, like thinking of replaying those Einstein matches, wouldn't be fair to anybody. Instead, appreciate what Frank Merrick and the people in FIRST have done, making the very best out of a terrible situation, and look forward to the more promising future. This incident has spurred FIRST into looking intimately at how the FMS works, giving rise to the potential for positive change that can make the years to come better than today. As much as it should, nothing will change from the 2012 FRC World Championship. Celebrate all twelve teams who were crowned Division Champions, and have faith that 2013 and beyond will be the best FRC seasons of all.

And it appears that we will never know the motives, or even the intent, of the "individual", so let's not pile on or ask unreasonable questions of or about the individual or his/her team. Going through the rest of your life without the opportunity to be involved with FIRST is already a tall order for someone who probably loved FIRST and poured as much into it as all of you have, but made one horrible mistake to bring it crashing down.

IanW

14-07-2012, 00:03

Unfortunately, while we on Chief Delphi like to maintain that one member of the team doesn't represent the entire team, it doesn't always work out like that, especially in cases of extreme circumstances. Now, you, or I, or anyone else on CD may believe the individual doesn't reflect the team, but there will always be people who believe otherwise. Any way you slice it (and I, personally, don't put much stock in this), people could look at it as "Well, this is the kind of people Team Such-and-Such has."

An individuals actions DOES, in fact, reflect the team they're associated with to some degree. Which is why I think it is best that the team and the person in question remain anonymous.

I think Leeland's (and others') reasons for allowing the person(s) who interfered with the matches on Einstein to remain unnamed are spot on. Based on my past observations of how the Chief Delphi community has handled situations where someone behaves inappropriately, it is INEVITABLE that both the person(s) and their team(s) would be metaphorically crucified. The response could have unintended consequences to the person(s) emotional state. In addition, the team would likely be unable to recover from the blow to its reputation. This is because, in my experience, the Chief Delphi community DOES NOT separate the actions of an individual from the actions of the team. Often times, I see members state that "your words/actions reflect upon your team."

For this reason, I would understand if the person(s) who engaged in the interference declined to make their identity known.

shawnz

14-07-2012, 00:14

On a different note, this is a brilliant example for teams of how to do Root Cause Analysis. It's also quite incredible to see what happens when you go over even the best robots with such a fine-toothed comb. Everyone can learn lessons from this document -- and even moreso when the FMS whitepaper is released! :)

Jon Stratis

14-07-2012, 01:07

Throughout the season, we saw a lot of posts and statements about "connection issues" at event and champs. Many people blamed FIRST through the entire season.

However, as this report shows, there are a LOT of issues that can affect a robot's ability to perform on the field, and many of them are caused (unknowingly) by the teams themselves. In the future, we should all keep in mind that these robots are very complicated machines, and there is rarely a single root cause for "connection issues".

FIRST did a great job with this report, and the sheer number of issues they discovered with the individual robots really shows how detailed they were in their investigation.

As for the individual who caused interference on Einstein... It's all been said at this point. The individual has been punished, and there isn't really anything else we can do about it. Part of Gracious Professionalism is not pointing fingers. When we find bugs at work, we don't ask "who wrote that section of code?" We ask "Who is the best, most knowledgeable person to work on fixing this bug?" As a community, lets move past the actions of the individual and show our support for the job FIRST has done.

Ian Curtis

14-07-2012, 01:42

...I have no other comments to make at this time.

I did exactly the same thing and spent a couple minutes thinking of just how awesome it would look to have towering Van de Graff generators over the old Manchester mill buildings... :D

This report is really awesome. Two thumbs up for FIRST because they really did their homework. Interesting a lot of the "Oh it can't possibly be team XXX" turned out to be not the case, and I think it will probably serve first well in the future as the community will probably not jump all over them at the first sign of trouble -- especially if high quality work like this can be expected in the future. It's also a side of engineering (failure analysis & technical report writing) that many FRC kids don't get to experience, so it's a interesting exercise from that point as well.

I'm just glad that the "hacker" is no longer in a position to ""inspire"" students.

techhelpbb

14-07-2012, 01:43

I disagree entirely. I don't believe anyone believes (or believed prior to Einstein) that the system is above flaw.

It's is in my experience over 17 years extremely common for people to assume that the field can not be the source of a problem. Often they are correct but that only makes it more troubling when they are not.

Take any system, no matter how well designed, and subject it to 60,000 ambitious folks all playing with it and see how secure it is.

No problem I do that every day literally. Only it's more than 60,000 people. I do computer security for a living as well as operating a few businesses that work with computing, electronics, and electrical. We have lots of security problems and we do our best to identify, qualify, quantify, document and offer resolution.

Sometimes we get solutions and sometimes we do not.
If we don't get resolution then we know where to look when the trouble starts.

This week's 'Yahoo' password hack displays just what happens when even the most competent network security is open for public interaction.

It's off topic but that's a bad example. Anyone competent wouldn't store a password in clear text in a database with that sort of exposure to risk. You pad, hash and salt (and it's very simple there are existing tools to do this for you). They obviously left this old stuff laying around without regard for the SQL injection attack that is all the vogue for XSS these days. In point of fact we've been using this as a wonderful example of exactly why I have a policy document for the developers to avoid this exact attack vector (they are only very lucky that it wasn't a black hat that went after them quietly). They were also not very forthcoming about the possibility of the scope of the breach as they have a XSS single sign on they implement. Worse...some people think it was 'Yahoo Voice' that was breached but there's another Yahoo service 'Yahoo Voices' (that's right it's one letter off and the reporters who have to handle the announcement are not keeping it straight). It's an example of everything you don't do if you value your security or your business before, during and after a breach.

I grant you they have lots of other security issues at Yahoo right now that I am well aware of as are plenty of others. Surely they are not the only company that fails to be vigilant or gets utterly complacent. I'm sure someone figured they were saving a dollar (and maybe they did).

However, not all breaches are equal. The more people know about a problem the more silly you will feel when you get nailed for it. FIRST's deauth vector is not new, Hack-A-Day exposed this very publicly last year and other sites well before that. All that was required to breach this? Download code.

Someone WILL find a way in. Google, Microsoft, and even the stock market have been subject to security invasions as well.

Perhaps the most effective hack is not a hack at all. Social engineering is the easiest and most effective hack because it hacks people. However, you don't differentiate you consider them all the same. Social engineering hacks are also why what you write next will not be nearly effective as you think:

I hate to say it, but in this situation security through obscurity is FIRST's best bet. The entire system needs to be removed from the consumer electronics spectrum that all these common tools are designed to work with. I.e. - standard a/b/g/n wireless needs to disappear. If this does not change and go to a proprietary system, I will 100% guarantee you WILL see this happen again.

Actually it doesn't matter what spectrum you use or how obscure. It's radio and it can be blocked cheaply and easily (though obviously illegally...but they gotta find you and prove it). As long as it's wireless denial of service will always be possible if you're willing to take enough risk as the attacker.

Obviously a band less frequently used will make it more obvious what you are doing. However once you commit to those frequencies without recourse they could hold you hostage long enough that the cost to continue will be extreme.

I don't actually disagree that they should move some of this from the bands where people accidentally could interfere with phones and such. I just don't think it matters as long as the field aspect is assured. So in that regard I think the field comm. specific stuff should be put somewhere and let anyone use WiFi for whatever they like. Let the users deal with the security issues, finding channels, and if you like give them a solution that ought to work in that regard but get out of the business of letting student written robot code interfere in field comms. The fields comm. stuff is generally unique to the competitions anyway, outside of the competitions WiFi is plenty workable.

StevenB

14-07-2012, 01:52

The attacker's motives don't strike me as dark and mysterious. The report specifically states that the individual contacted FIRST and explained what they had done. If they hadn't, we would probably still be speculating. Were their actions in St. Louis wrong, foolish, and harmful? Yes. But I hesitate to say they were malicious.

Imagine this: You're sitting in the stands at your regional, watching a match, and just out of curiosity, you try to connect to the FMS from your phone. The phone asks for the WPA key. "Of course," you think to yourself and give up on this little experiment. But you punch in a random password anyway, and to your horror, the robot on the field suddenly stops.

You try to tell the FTA, but he brushes you off. Lots of robots die for lots of reasons, and there's absolutely no reason for a failed authentication attempt to take out a connection.

Now what?

From reading ChiefDelphi, it's clear to you that this is probably happening all over the country. FIRST HQ seems to be ignoring the control system issues and is blaming the teams instead, and it makes you angry. (Go back and read some threads from March - this sentiment abounds!) Championships are going to be a mess. It's almost certain that others are going to discover the issue, and they will probably use it in a malicious way to gain an advantage in the competition.

So, in anger and wannabe-heroism, you do the only thing you can think of to get FIRST to listen...

Was this the wrong way to address the issue? Absolutely. But given the openness, sincerity, and determination we've seen from FIRST in recent months, I wonder if the attacker succeeded.

This investigation uncovered a treasure trove of software bugs and electromechanical faults, and I hope we will all build better robots next year because of it. It showed that problems, both incredibly complex and completely stupid, will happen to all of us. Major thanks to FIRST for such a thorough investigation, and I'm looking forward to the improvements next season!

Alexa Stott

14-07-2012, 02:06

Imagine this: You're sitting in the stands at your regional, watching a match, and just out of curiosity, you try to connect to the FMS from your phone. The phone asks for the WPA key. "Of course," you think to yourself and give up on this little experiment. But you punch in a random password anyway, and to your horror, the robot on the field suddenly stops.

You try to tell the FTA, but he brushes you off. Lots of robots die for lots of reasons, and there's absolutely no reason for a failed authentication attempt to take out a connection.

I'll just leave this here.
This individual was observed to be pulling up a screen which contained the team numbers of the six teams currently competing, selecting a team, and then rapidly typing in text[...]The individual using the Galaxy Nexus phone was observed to be rapidly repeating this process until shortly before the end of each match.

Anyway, this will be my last post on the matter. The intentions can never truly be known.

Chris is me

14-07-2012, 02:18

I'll just leave this here.

He was speculating as to how he / she discovered the bug - not how he / she executed it in St. Louis.

SM987

14-07-2012, 02:21

So, in anger...

Pretty close to malice.

That situation is of course a hypothetical one, but we can't justify or glorify the hackers actions because of the "silver lining" this investigation was.

techhelpbb

14-07-2012, 02:22

As quoted from elsewhere: "This individual was observed to be pulling up a screen which contained the team numbers of the six teams currently competing, selecting a team, and then rapidly typing in text[...]The individual using the Galaxy Nexus phone was observed to be rapidly repeating this process until shortly before the end of each match."

This is going to sound strange but scouting app?

After all even the Einstein teams will compete off season.

Several teams have applications like this in the Google Play store that anyone can download.

dodar

14-07-2012, 02:51

This is going to sound strange but scouting app?

After all even the Einstein teams will compete off season.

Several teams have applications like this in the Google Play store that anyone can download.

That have live match information? Not even the FIRST database has live match-by-match data. It couldnt be anything else other than the FMS system this person was looking at.

Lil' Lavery

14-07-2012, 04:32

The information contained in the report and the information omitted from the report was selected for a reason. Attempting to read between the lines will only grant speculative conclusions. Wild speculation is exactly what this report was attempting to mitigate.

The "Root Cause Conclusions" table on page 20 of the report lists fifteen instances of command response failures. Six of those instances have nothing to do with the "Failed Client Authentication" issue. Only one of the fifteen instances is "confirmed" to be because of the "Failed Client Authentication" issue. Eight matches were ran on Einstein, each with six teams participating. That's 48 potential opportunities for command response failures. Setting aside the FCA issue, 12.5-14.6% (depending on 2056's root cause in SF2-1R) of the opportunities still manifested command response failures.

Regardless of whether or not the root cause was the fault of the field or the team, a system with a 12.5% failure rate among the elite participants at the end of the season is simply not acceptable. This report was not about the practice day at an early season regional, but the finals of the championship event. A vast majority of this report is not focused on the Failed Client Authentication issue, but the numerous other potential points of failure in the system both on the robot and FMS sides of the equation. I'm glad FIRST is taking steps to try and improve this system and remedy the issues highlighted in this report, and this is far more important to me than debating the motives and proper punishment of an individual.

MikeE

14-07-2012, 04:46

Great report and worth waiting a few weeks for. It's easy to become cynical about the organization, but this is an example of FIRST doing things right.

Quick thoughts:

Disappointed but not really surprised that there was intentional interference. It's been a huge risk and getting worse now that many participants are carrying at least one 801.11n capable device in their pocket.

Surprised that even the top-tier teams had so many electrical and software problems. I've seen a variety of issues among the regular teams at regionals and this does all suggest that control system support is inadequate even now that we're at the end of it's 4th season.

The overall programming model is not well understood, as evidenced by the multiple issues with *Continuous methods. Better documentation here will be very welcome.

Similarly, two thumbs up for the promised FMS whitepaper.

Unfortunately the "individual" will become known. Not that many people are on/next to Einstein during the finals, and this community is too curious/dogged for the information to remain hidden.

Don Wright

14-07-2012, 05:16

The problem with not naming the person/team is that the entire FIRST community is now guessing who it is...and probably coming up its a few ideas (some come immediately to mind for me). Is that a better alternative...to have people guessing forever? Or wait until next season when the person, who might be a popular person in FIRST suddenly isn't involved anymore?

Billfred

14-07-2012, 07:12

To anyone still showing a bit of mercy for "the individual", the standard quote from Woodie came to mind (source (http://www.usfirst.org/uploadedFiles/Community/FRC/FRC_Documents_and_Updates/Reg_Planning_Guide/RPG_Introduction.pdf)):

“Obviously it would not make sense to endorse ‘asinine professionalism’ or ‘gracious incompetence.’ It is, however, completely consistent with the FIRST spirit to encourage doing high quality well informed work in a manner that leaves everyone feeling valued. Gracious professionalism seems to be a good descriptor for part of the ethos of FIRST. It is part of what makes FIRST different and wonderful.
Gracious professionalism has purposefully been left somewhat undefined because it can and should mean different things to each of us. We can, however, outline some of its possible meanings. Gracious attitudes and behaviors are win-win. Gracious folks respect others and let that respect show in their action. Professionals possess special knowledge and are trusted by society to use that knowledge responsibly. Thus, gracious professionals make a valued contribution in a manner pleasing to others and to themselves.
In FIRST, one of the most straightforward interpretations of gracious professionalism is that we learn and compete like crazy, but treat one another with respect and kindness in the process. We try to avoid leaving anyone feeling like they are losers. No chest thumping barbarian tough talk, but no sticky sweet platitudes either. Knowledge, pride and empathy comfortably blended.
Understanding that gracious professionalism works is not rocket science. It is, however, missing in too many activities. At FIRST it is alive and well. Please help us take care of it.

The 2012 FRC manual leaves out (http://frc-manual.usfirst.org/viewItem/3#1.2) the first and last paragraphs (note to Frank for next year...), but to me it forms the most basic code of FIRST's programs. I'd have a hard time buying an argument that the individual's actions are compatible with what's above.

shawnz

14-07-2012, 09:25

That have live match information? Not even the FIRST database has live match-by-match data. It couldnt be anything else other than the FMS system this person was looking at.

Perhaps he was *entering* the live match data. This is how our team's scouting system works currently, and I know a number of teams use similar techniques.

I'd have a hard time buying an argument that the individual's actions are compatible with what's above.

I should hope nobody is suggesting it is! I think the argument here was whether the anonymity granted to the person was justified or not, rather than the blacklisting.

iVanDuzer

14-07-2012, 09:56

If you honestly believe that you can draw parallels between not attempting the coop bridge with very very good teams and deliberately sabotaging the premier FRC matches, I don't even know what to say...

If you think the situation at GTRE was as simple as "not cooping with elite teams" then sorry, you need to get your head checked. The problem at GTRE was teams trying to influence the coop points of matches they were not involved in through conspiracy and, in some case involving rookies, bullying ("Nobody will like you if you balance with these teams"). Not cooping is a valid strategy. Trying to convince other teams not to coop is not. FIRST made this clear at the Week 3 events through a Team Update (IIRC), an emailed team notice, and Dean's own comments at the Montreal Regional.

Sidenote: I don't believe the events on Einstein are related to those at GTRE. The GTRE issues could have occurred anywhere where there are two elite teams that stand heads and shoulders above even the 3rd best robot. Imagine, for example, a Michigan where the only powerhouses are 469 and 67. Or a California with just 254 and 1717. These two teams pair up and dominate year after year. Of course this builds up some level of animosity. This isn't a Canada issue; Canada just so happens to be where this issue surfaced.

_____
As for 1114's official statement, I agree wholeheartedly. Mistakes were made, purposeful interference happened. Don't hide it. The great thing about 1114 asking for a public apology is that, if they were in the same situation, they would apologize in a heartbeat. Reputation can be rebuilt, but suspicion can't be dissolved without clear answers.

We are constantly told that the Einstein teams are the Best of the Best, not only because they have the greatest robots/strategic minds of that year, but because they are class acts. Apologizing admits that there may be something amiss in team culture, or that the individual was a bad apple but wasn't originally thought to be. Admittance is a very good first step in the right direction. I for one would applaud any team that came forward to admit that a member committed sabotage and also said they are working to fix the problem in their own team.

Keep the individual anonymous, and I'm ok with that. But it's the team's responsibility to make sure that EVERY team member "gets FIRST." I think it's a good thing that an individual's actions impact the reputation of a team. It forces a team to look at itself and intentionally weed out bad apples and turn them into glorious, shiny red ones with no worms. Asking for an apology for being unable to do this, in my opinion, is justified.

BornaE

14-07-2012, 10:00

Any laptop with a wireless can would show the list of teams on the field since the team number is used as the SSID.

Perhaps he was *entering* the live match data. This is how our team's scouting system works currently, and I know a number of teams use similar techniques.

I should hope nobody is suggesting it is! I think the argument here was whether the anonymity granted to the person was justified or not, rather than the blacklisting.

techhelpbb

14-07-2012, 10:00

What concerns me the most is that someone who finds out who or thinks they did will think it is their job to play hero.

All it takes is someone to start writing letters or going after this person professionally outside of the parameters of what FIRST deems required.

FIRST is the biggest victim here. Not even the teams. There is a valid tangible dollar amount of damage done here. Beyond FIRST these teams have no one to turn to ask for compensation. I really think it is FIRST who should decide how they dish out the punishment and we should respect that. If we do anything more I think you are starting to stray into a place where you put your personal concerns above FIRST.

I personally have seen what happens when just a few people decide they know better how something should work and will take it upon themselves to go that extra step to have things a way they prefer. It rarely ends up the way they thought because they might not speak for the majority who has a less personal investment in their more personal motives.

I can only point out that we as members of a community are dedicated to gracious professionalism and activity to support that which itself represents that ideal. If this person became known and a few people decided to go even a little too far we as a community would be tarnished but it would then be a self fulfilling cycle.

Let us move forward into the future. Let us additionally focus this extra outrage on the continuing risk that someone might know someone that might try to do this again. If you know that someone is specifically intending to interfere in match operations and has taken action to actually make that interference happen please tell someone. Much of the annoyance with this could have been mitigated if someone tapped this person on the shoulder sooner and discussed it. Security is everyone's responsibility. Not just AirTight or FIRST. To me it really is the same as any number of more common safety hazards that crop up all the time.

Libby K

14-07-2012, 10:02

Okay, it's that time again where Libby chimes in with a seemingly unpopular opinion.
I want to stress that I am not trying to be accusatory, and I'm also not speaking for any of my teams, for FIRST, or for my family.
This is all me, and only me.

As much as I can, I use the concept of gracious professionalism as an internal yardstick. However, I have a really hard time believing anyone would think that the correct way of carrying yourself when in possession of an issue like this is to disrupt the climax of FIRST's largest, most-anticipated, most-covered event.

If you'd like to make a point, to an FTA, email someone at FIRST. Don't ruin the biggest event of the year and make a fool (or, you know, a criminal) out of yourself in the process.

I personally hope for the team to come forward. If this individual were acting alone, and the team can honestly say they didn't know about it, then the team should be able to say "Yes, Jimmy (or Susie) McHacker was a part of our team, we didn't know at the time what was going on, but our team didn't condone the behavior and they are no longer on the team because of their action. We're really sorry." (Obviously, being banned from the team is no longer necessary since they're barred from FIRST, but that's not the point.) Yes, there will still be people in this community that will reflect the actions of the individual onto the team... but those people would be wrong, and we'd know that.

...the team (or whoever on the team is aware of the individual) should release the information and acknowledge that it was an individual, not a team effort. Imagine the reputation of the team if the information came out sometime in the future by another source - it would not look good. If the team came out it would be seen as a gracious step forward...

Should the team then remain anonymous to avoid potential scrutiny? Again, what if the information was released by another source? Coming forward now would be the best way to avoid scrutiny at a greater scale.

^Exactly. It looks far better for the team to come forward now than have it disclosed at a later date by another team/individual... and knowing our community, that's going to happen at some point.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

You're taking an incredible moral high road, and saying that this person feels sorry. Good for you for believing in people, but what if they don't?! Unfortunately I've seen plenty of attitudes that say "I'm GLAD Einstein screwed up, because FIRST sucks and they had it coming". Paraphrased from many emails/FB comments/tweets/what have you, but that's the sentiment.

For this one person who deliberately tried to sabotage the event, there are literally hundreds of thousands of people who would condone in. Lets remember that one rogue person shouldn't be regarded as a significant change in FIRST culture.

I think you meant 'condemn' and not 'condone' but your point shines through. I sincerely hope this individual was acting alone and that this does not reflect a change in FIRST culture.

--

Regardless of the individual I want to thank the wonderful people at FIRST HQ, the volunteers that helped with the testing process, and, of course, the Einstein teams. This is the biggest disaster FIRST has ever seen and you all handled it with class and professionalism. Thank you for being shining examples of what FIRST teams and participants should be, even during the bad times.

I'll leave you with a Woodie quote...
"Understanding that gracious professionalism works is not rocket science. It is, however, missing in too many activities. At FIRST it is alive and well. Please help us take care of it."

Thank you, all, for helping us take care of it.

Nate Laverdure

14-07-2012, 10:35

Wow, I just found this:
14 Jul 2012 10:00 EST
FOR IMMEDIATE RELEASE
SAMSUNG TO WITHDRAW FROM AMERICAN MARKET FOLLOWING ATTACK BY ROGUE DEVICE

The actions of the individual Samsung Galaxy Nexus device that deliberately disrupted the 28 Apr 2012 Einstein matches were reprehensible and unjustifiable. Samsung vehemently denies any knowledge of, or participation in, these activities. These actions were undertaken without the knowledge and consent of Samsung and in no way should be considered to represent what is considered acceptable for a Samsung device.

Samsung will make all possible efforts to identify and bring to justice the specific device responsible for the actions of 28 Apr. In addition, Samsung is committed to prevent all future occurrences of this kind of attack by rogue devices. For this reason, Samsung and its corporate partners are taking immediate action to shut down all US and Canadian manufacturing and sales activity, followed by a phased withdrawal from the North American market. Additional details will be forthcoming.

For information contact:
Samsung Electronics America
85 Challenger Road
Ridgefield Park, NJ
I personally applaud this bold move from Samsung. They've understood that we won't be able to move forward as a community until the individual Galaxy Nexus phone, and Samsung as a whole, is held to account.

shawnz

14-07-2012, 10:42

Wow, I just found this:

[...]

I personally applaud this bold move from Samsung. They've understood that we won't be able to move forward as a community until the individual Galaxy Nexus phone, and Samsung as a whole, is held to account.

Too subtle?

techhelpbb

14-07-2012, 10:46

Unfortunately I've seen plenty of attitudes that say "I'm GLAD Einstein screwed up, because FIRST sucks and they had it coming". Paraphrased from many emails/FB comments/tweets/what have you, but that's the sentiment.

I just wanna say that I personally voiced concerns about several issues that did apparently impact Einstein. I was aware of deauth and what it could do. It's part of my daily responsibility to educate myself in security issues and I knew about that for more than a year...though I never imagined that someone would intentionally do that (assuming they did it intentionally which I think we may never know).

My daily activities involve risk analysis. The shear amount of risk means that more often than mitigate all those risks personally (which in a way I have done with by prototyping some things I've offered to FIRST) I spend time writing up that risk and making it clear to other people that they accept what I deem as those risks by not taking some mitigating action (whether it's the one I recommend or not is up to them).

I've been a vocal advocate that the risks for an outcome like Einstein in robot power quality have been present for too long (for years). That those risks having been under communicated or under addressed could be a real problem and this report somewhat vindicates that point. FIRST is taking the position that they'll educate but the core problem remains. We build robots that crash into things, are moved frequently while not under power and the same is true for the field. Things are going to break. It doesn't matter how much you write reports people need the tools to diagnose those issues within the time frame the competition offers.

I tried to offer FIRST assistance at Einstein via communications in this forum and later via communications up to and including requests in the official forum. As a majority the risks were accepted that's not my job to do a little dance of pride about that when what I worried about happened.

It is however part of the healing process to make it clear in the aftermath that we can't ignore the underlying process that accepted this risk and insure that in the future we all more fully acknowledge the risks going in.

There are tragic moments in my life where I have pointed out risks to people and a great number of people died including dozens of friends of mine because they took a risk I deemed as reckless and complacent. You can stand there in shock and worry about laying blame or use the failed responsibilities as a tool to honor that which was lost with practical goals in mind.

I just want to make it extremely clear. I personally get no joy from being right when something bad happens I may have warned about. It reminds me every day that people often set their priorities in ways that take risks and don't know what to do once the risk is proven with consequences.

FIRST has expended a great effort with this report. However, this is hardly the end of it. This demands that FIRST consider ways to make sure that power quality issues can be analyzed with in the time frames they desire to operate. It further demands they more actively consider the security risks to their communications systems moving forward in the grander sense beyond this one deauth issue. To do anything less is to ignore the lesson cause and effect is offering.

What isn't apparent from this report because it hyper focuses on Einstein is how much of this happened years before and how much of it happened into the seeding up to Einstein. The fact is it is entirely possible that the whole of the competition was shaped by deauth and power quality issues in no small way.

Ether

14-07-2012, 10:48

techhelpbb

14-07-2012, 10:51

Wow, I just found this:

I personally applaud this bold move from Samsung. They've understood that we won't be able to move forward as a community until the individual Galaxy Nexus phone, and Samsung as a whole, is held to account.

Considering that Samsung just release the Samsung S3 I am holding in my hand forgive me if I doubt the validity of this statement in the sense it could be taken.

Such a move would financially utterly destroy their company and in point of fact cause massive financial damage to the manufacturers that support them.

Besides they are already banned from selling the Nexus:
http://www.businessweek.com/news/2012-06-29/apple-wins-order-blocking-u-dot-s-dot-sales-of-samsung-s-nexus

O'Sancheski

14-07-2012, 10:52

Wow, I just found this:

I personally applaud this bold move from Samsung. They've understood that we won't be able to move forward as a community until the individual Galaxy Nexus phone, and Samsung as a whole, is held to account.

Do you have the source for this? Because to my understanding, Apple sued Samsung and told them to stop producing the Nexus.

Nate Laverdure

14-07-2012, 10:56

Too subtle?
Yeah, I guess. My fault.

All: Disregard the above, maybe I'll try again later.

IndySam

14-07-2012, 10:57

I am constantly amazed at how sarcasm challenged many people are Nate. :)

shawnz

14-07-2012, 10:57

It's a joke, guys. I think he was trying to make an analogy between incriminating individuals and incriminating a whole team. It's not a great analogy for a lot of reasons, though.

EDIT: Late

techhelpbb

14-07-2012, 10:59

Do you have the source for this? Because to my understanding, Apple sued Samsung and told them to stop producing the Nexus.

To be clear they can't stop the production of the Nexus or the Galaxy Tab 10.1. I can produce anything I like and so can they (assuming it's not something subject to military restriction). They just can't sell them in the American market.

I can in point of fact still buy the Nexus, the Galaxy Tab 10.1 (GameStop has them) and of course the Galaxy Tab2 10.1 (which I just returned the other day for specific technical reasons).

The court did not order them to even empty the distribution of the product. Merely to stop offering to sell new stock until the matter is resolved.

Adding:

Sorry but that's sort of not a funny joke. You do realize that I have relatives that supply parts to Samsung and I know other FIRST members that if they read that would be wondering where their paycheck will come from. It's a tough economy out there, and this is a popular topic for people to read. Please think before you spoof your joke might not be funny if someone else panics. Misunderstandings like that can themselves shift the value of a stock.

LeelandS

14-07-2012, 11:02

You're taking an incredible moral high road, and saying that this person feels sorry. Good for you for believing in people, but what if they don't?! Unfortunately I've seen plenty of attitudes that say "I'm GLAD Einstein screwed up, because FIRST sucks and they had it coming". Paraphrased from many emails/FB comments/tweets/what have you, but that's the sentiment.

Maybe my moral road borders on "So high, the oxygen is too thin up there." So I understand what you mean. But for right now, it's "What if..." vs. "What if...". We don't know the person's motives. We don't know how they feel about their actions. Maybe they really do feel bad, maybe there is no remorse what-so-ever. We just don't know.

In my personal views, any way we slice it, all involved parties remaining anonymous just seems like the best route. Realistically, what does the person coming public do for the FIRST community and, possibly more importantly, the 12 Einstein teams. Obviously 1114 wants the team/individual to come forward, but what obligation does the team/individual have? Yes, it would be the "Bigger man" thing to come forward. But they don't need to come out publicly to apologize. I would recommend that the person in question write a personal letter to each of the 12 Einstein teams, FIRST, and MAYBE an anonymous letter addressing the FIRST community. I don't think we have any business knowing, personally, who the person or team is.

That's just my view. Maybe I'm being to forgiving or careless. Oh well.

techhelpbb

14-07-2012, 11:13

Maybe my moral road borders on "So high, the oxygen is too thin up there." So I understand what you mean. But for right now, it's "What if..." vs. "What if...". We don't know the person's motives. We don't know how they feel about their actions. Maybe they really do feel bad, maybe there is no remorse what-so-ever. We just don't know.

In my personal views, any way we slice it, all involved parties remaining anonymous just seems like the best route. Realistically, what does the person coming public do for the FIRST community and, possibly more importantly, the 12 Einstein teams. Obviously 1114 wants the team/individual to come forward, but what obligation does the team/individual have? Yes, it would be the "Bigger man" thing to come forward. But they don't need to come out publicly to apologize. I would recommend that the person in question write a personal letter to each of the 12 Einstein teams, FIRST, and MAYBE an anonymous letter addressing the FIRST community. I don't think we have any business knowing, personally, who the person or team is.

That's just my view. Maybe I'm being to forgiving or careless. Oh well.

The real risk is statistical. If we know the team by admission then the number of suspects for the individual is reduced to below 300 people. Then all you need to do is cull the list of former team members from the last year and you're down below 50 people who could have done the deed. That's actually a small enough number that a dogged investigator will find the person in question shortly and with specifics.

I wouldn't be surprised if FIRST has bound them legally to silence it would be the smart move.

Greg McKaskle

14-07-2012, 11:40

My own personal wish is that this situation will serve as a somber discussion point within each team. What would the correct response be if YOU found this or a similar issue? What if ... 's are a very valuable teaching tool for life, and the FIRST community is a safe harbor for learning so many of those life tools.

The fact that this statistically unlikely combination of wifi components left the door to the FIRST field partially unlocked doesn't make it that unique. A lost wallet, an unlocked or easily bypassed physical door, an online account with a weak password, all of these take place day-to-day and are worth considering and discussing along with the Einstein vulnerability. All systems have weaknesses, and for me it is cultural expectations more than the strengths of the locks that provide a sense of security in my day to day life. I am not intending to make excuses for the field issues. Locks and alarms are certainly a necessary part of the solution, but what really counts is how people behave when the lock is missing and there are no witnesses.

Relatively speaking, I'm still new to FIRST, and I'm still amazingly proud of the impact it has on people. The level of trust and generosity displayed within the community is practically unparalleled. And yet, it will not maintain itself. It can be improved. I hope that the ultimate outcome of this unfortunate situation is that thousands of individuals consider their own actions in this and similar what if ... scenarios and use it as motivation for self-improvement.

Greg McKaskle

Steven Donow

14-07-2012, 11:43

Wow, I just found this:

I personally applaud this bold move from Samsung. They've understood that we won't be able to move forward as a community until the individual Galaxy Nexus phone, and Samsung as a whole, is held to account.

My mom's 3 day old Galaxy S3 randomly died this morning. This makes sense.

Anyway, I don't think the team nor the individual needs to come forward. The team could easily come out and say that they didn't know about it/weren't involved with it, but whose to say that's the truth? I could easily say right now, "I did it.", and if I phrased it as an eloquent apology, people would believe me.*

Also, just a random idea here, whose to say the team knows about it? Maybe the interferer didn't tell his/her team why they were leaaving, and just treated it as a retirement or something.

Just throwing out some possible scenarios here.

*For the record, no, that was not a confession, wasn't a smartphone owner when I was at CMP, and I was the only one on my team there.

Racer26

14-07-2012, 11:44

Two things I noticed:

The letter from Jon Dudas, and the report itself state that the culprit of the intentional interference was not a member of one of the winning teams. If the team associated with this individual was not one of the 12 Einstein teams, why wouldn't Jon simply say that? It's possible Jon simply overlooked this, but the wording is interesting.

It also said that the culprit was fieldside. Trying to speak with the FTA during the confusion.

As I recall (I've only been to CMP once), the number of people allowed fieldside at Einstein would be restricted to the drive teams of those teams participating on Einstein, plus some volunteers.

That leaves a VERY small number indeed.

I definitely agree with others that the individual in question likely WILL be found out in relatively short order.

I also suspect that the 12 teams (or at least their representatives present at the investigation weekend) probably know, and were asked not to talk.

Hjelstrom

14-07-2012, 11:45

As for the individual who caused interference on Einstein... It's all been said at this point. The individual has been punished, and there isn't really anything else we can do about it. Part of Gracious Professionalism is not pointing fingers. When we find bugs at work, we don't ask "who wrote that section of code?" We ask "Who is the best, most knowledgeable person to work on fixing this bug?" As a community, lets move past the actions of the individual and show our support for the job FIRST has done.

Do people at your job purposely hide bugs in other peoples code to get their rivals fired? I work as a software engineer too and take the exact same approach to bugs (they are unintentional mistakes after all). That analogy does not work at all here.

Look, there is more to this story. We actually do know exactly who did this and we know more about their motive than you'd think. No it wasn't curiousity or accidental. We had eye-witnesses to some suspicious behavior right on Einstein. We took pictures of the person. We told the FTA right on the spot. We even told our opponent 1114. We didn't put the whole puzzle together until New Hampshire (i.e. we didn't fully understand what we were seeing at the time) but now we know exactly what happened and who did it. This person was cheating plain and simple. It was definitely not a scouting app. Right now if we just wanted to crucify the team we could. That is not the purpose of us asking them to come forward.

Here is the reason we ask: We've already seen other specific teams mentioned and considered as potentially involved. I've also seen people minimizing what happened and even some saying FIRST didn't really figure anything out! Go ahead, how many of you have a suspect in mind! This is not a good situation to leave this community in. I honestly believed the team would have come forward by now to bring closure to this. It makes me extremely upset that they haven't because by their secrecy it damages the other teams that were there. We're trying to be *gracious* and *professional* by allowing them time to do the right thing rather than throwing the book at them.

For the people trying to rationalize or minimize this, you don't know the whole story yet. I believe that every case of interference in the report was in fact interference and if it comes to it we can present a very convincing argument why.

Racer26

14-07-2012, 11:53

@Hjelstrom: As I understand it, being one of the people who was AT the investigation weekend, you may wish to be careful how much you say, especially when its things you don't have proof of.

Hjelstrom

14-07-2012, 12:05

Fair enough. I removed the speculation at the end.

Racer26

14-07-2012, 12:06

I'm not entirely sure about that incident, but I had a sneaking suspicion even before Worlds that we were somewhat vulnerable to dropouts when being hit near the battery. It could have been that, considering it coincided with a collision right there with the corner of another robot.

Plus, it was only once, and the NI experts in the pits actually reviewed our code and found it to be fully functional and efficient enough (~50% CPU usage average).

Video here (http://www.youtube.com/watch?v=IMPc8myQAn8&feature=relmfu)

RSL is clearly flashing while it's dead, suggesting the cRIO is still running. Symptoms match what the report says happens with FCA.

Also, 1717 in this video (http://www.youtube.com/watch?v=Vp6EQYAg1BI). Symptoms match again.

Hjelstrom

14-07-2012, 12:42

Video here (http://www.youtube.com/watch?v=IMPc8myQAn8&feature=relmfu)

RSL is clearly flashing while it's dead, suggesting the cRIO is still running. Symptoms match what the report says happens with FCA.

Also, 1717 in this video (http://www.youtube.com/watch?v=Vp6EQYAg1BI). Symptoms match again.

Those videos give me chills. Wow.

smurfgirl

14-07-2012, 13:03

Wow, I'm completely shocked by what I read. I can't imagine what drove someone to do that.

I think FIRST handled this the best they could, though.

Pat Hart

14-07-2012, 13:04

One question to ask when demanding the person to come forward and apologize is "Could I forgive this person?" If the answer is no than perhaps the only reason you want to know the name is so that you can string this person up. Which leads to more ungracious and unprofessional behavior.

Only if the answer is yes will there be anything productive coming from a public apology.

What the person did is terrible and there should be just punishment and consequences, but this person does not need to be forever hated or despised.

Holtzman

14-07-2012, 13:15

First off, I must thank FIRST for its thorough investigation. They did a pretty good job of keeping us in the loop when it was appropriate. Frank, Matt, Ryan, and Kevin were all very supportive through this difficult time for us.

To all the people who have thrown in their two cents, I ask you to consider this. You have absolutely no idea what we have been through this season.

I had to stand in front of a room full of my students, parents, and our sponsors and explain to them that after we had worked so hard, and come so far, that someone felt the need to intentionally disable our robot on Einstein and prevent us from even competing on the world’s biggest stage. How many of you can say the same thing?

I attended the Einstein Weekend with Eric Mech, one of our graduating seniors. Eric never left the school before 9pm this season. He was our lead programmer and poured his heart and soul into the team. All the teams on Einstein were brought together and told about the FCA attacks at a meeting on Sunday morning of the Einstein weekend. We were told that FIRST had evidence that our robot was specifically targeted and disabled. I raised my hand and asked what evidence they had, and then Frank Merrick told me that the person had openly admitted to it. I got up and had to leave the room. I was furious. Eric stayed, and handled the news a lot better than I did. He graduates this year not knowing of what we could have achieved.

Would we have won on Einstein? Maybe, maybe not. 180, 25, and 16 were a very strong and deserving alliance. If we had lost to them, or to 987, 233, and 207 on even terms, we would have held our heads high, and congratulated the better alliance. I'm proud to call many members of these teams friends. But the fact is we were denied the chance to even compete by one incompetent jerk.

Trips to Einstein don’t come around very often. Even if you build the best robot in the world, you still need a darn good bit of luck to make it there. I just hope we get the chance to go back and play on even terms.

We in no way blame the entire team for the actions of this individual, but do feel they should stand up and acknowledge that a member of their team was responsible for the FCA attacks on multiple Einstein teams, and potentially others at the Championship and other events. We will give them time to do this properly while being respectful to the innocent members of their team.

On a separate note, the events of GTR-E have absolutely nothing to do with the events of Einstein. All the Canadian teams at the championship were hugely supportive, and we thank them all for that. We are looking forward to playing with 781, 772, and 907 as well as the rest of The Eh Team at IRI next weekend.

I will close with one rather ironic story. I had a gentlemen come up to me on Thursday at The Championship, and thank me for a recent post I had made on Chief Delphi on an unrelated topic. He told me that 2056 is usually very quiet, and likes to let their robot speak for them. Under normal circumstances, this is the way we like to operate. Unfortunately, it’s difficult to let our robot do the talking when it sits dead on the field.

Alan Anderson

14-07-2012, 13:27

FIRST's deauth vector is not new, Hack-A-Day exposed this very publicly last year and other sites well before that. All that was required to breach this? Download code.

I don't think you understand what the actual problem was. The system is indeed vulnerable to a deauthentication flood, or even a fast trickle. However, there were no such attempts detected, and there is no evidence to suggest that any occurred. The testing did show that it was possible to disrupt the connection without triggering a warning, so the detection parameters need to be tweaked to something more appropriate to the FRC use case. Still, it doesn't look like this was something that actually happened during competition.

The confirmed problem was instead an unknown and unexpected bug in the access point firmware that broke the existing connection when another client tried to authenticate and failed. Nothing special needs to be downloaded in order to cause this bug to be expressed.

JB987

14-07-2012, 13:38

We in no way blame the entire team for the actions of this individual, but do feel they should stand up and acknowledge that a member of their team was responsible for the FCA attacks on multiple Einstein teams, and potentially others at the Championship and other events. We will give them time to do properly while being respectful to the innocent members of their team.

+1

apalrd

14-07-2012, 13:42

While we're sharing dead robot videos, how about

-67 in MSC SF1-2 http://youtu.be/ZbTHqBdvgJc - Clearly dead for the remainder of the match, blinking RSL.

-67 in MSC F-2 - This to me shows symptoms that could also be described as a loose ethernet cable. Team 67 dies in the key, and recovers when they are hit by team 469 (~15secs dead based on video). They then die later in the match, near the opponents bridge, and do not recover. I will share it anyway. http://youtu.be/CDLzSMPyYsc
-We died as well in the SF's under similar circumstances. I will not share the video.

I (as a member of the drive team) was by the field during all of those matches. In the all cases, the FTA's blamed the issue on the radio, and team 67 replaced their radio with a spare after SF1-2. I talked with team 67 (and team 469, their partners) personally after/during this, and they very thoroughly looked over their machine and found nothing (to my knowledge).

As to all of my earlier rants on the control system, I have two points that I was trying to make:
-The system is too complicated or difficult to setup if so many of the highest level teams cannot get it right.
-The system, while vulnerable to an unknown bug, was also vulnerable to a known, relatively simple bug (deauth) and there was nothing in place to detect or log this properly.

Ryan Dognaux

14-07-2012, 13:42

I attended the Einstein Weekend with Eric Mech, one of our graduating seniors. Eric never left the school before 9pm this season. He was our lead programmer and poured his heart and soul into the team. All the teams on Einstein were brought together and told about the FCA attacks at a meeting on Sunday morning of the Einstein weekend. We were told that FIRST had evidence that our robot was specifically targeted and disabled. I raised my hand and asked what evidence they had, and then Frank Merrick told me that the person had openly admitted to it. I got up and had to leave the room. I was furious. Eric stayed, and handled the news a lot better than I did. He graduates this year not knowing of what we could have achieved.

Thank you for posting this. This personal account is just an example of how one person and one team was affected by this, and it's heartbreaking. The person responsible for this owes, at the very least, a personal apology to every single team that made it to Einstein this year. You not only made FIRST look bad, you took away something that in all likely-hood these students, mentors, teachers, parents & sponsors will never get back - a fair shot at winning the 2012 Championship.

I really wish the reason behind all the issues were purely related to technological failures, because we can fix those. This is a people issue. People are much harder to fix. I'm encouraged after reading the report that the technology-aspect of this competition will improve for next year's season. Let's strive to improve as people and as a community as well. No one should have to go through what these teams went through ever again.

Alan Anderson

14-07-2012, 13:55

One question to ask when demanding the person to come forward and apologize is "Could I forgive this person?"

What was done is unforgivable.

What the person did is terrible and there should be just punishment and consequences, but this person does not need to be forever hated or despised.

I do not believe everlasting hate is justified, but I see nothing wrong with the person being forever shunned.

Ekcrbe

14-07-2012, 14:22

EricH

14-07-2012, 14:25

For everyone who is saying that the team should come forwards:

Do you REALLY want that team's members to be known as "the team that ruined Einstein 2012" for years down the road? In their careers, even?

Whether or not that is your intent, that is what will happen. The team will gain that reputation, like it or not, if they come forwards. No amount of "This was an individual, not the team, we as a team don't go that route" and "This individual is no longer part of our team" will stop it. If it was one of your students or mentors, and you put forth your team as their former team, could you handle that stigma for years to come, in addition to any other mis-informed and unfair ones you may already carry or aquire in the future? Or would you or your sponsors simply cut the team?

If the individual were to announce himself or herself by issuing an apology, while leaving team identity out of it, that's another matter. In that case, one person gets the stigma--again, they have to deal with it. But again, certain curious ones may probe deeper and discover the team. See above paragraph.

If this act was truly the act of a lone person, operating without the consent or knowlege of the team, then the best way to keep the team from being tainted is to keep the person's identity--and that of their team--a secret known only to FRC staff, and as few of those as possible. If, on the other hand, the team was involved in some way, then there are other measures that FIRST can impose, which they have not.

With that said: The report, as a whole, shows a thorough investigation of the Einstein problems. It was never meant to investigate all the comm problems in FRC this year--such a study would take a full year. However, it does provide a path forwards to eliminating comm problems, and better diagnostics in the future. I like the report, both as a very well-written piece of technical writing and as an explanation of what happened on Einstein, and also as a guide to places to start looking for other issues that may be plaguing teams.

ratdude747

14-07-2012, 14:31

A tad late to the party here, but here's my $0.02:

First, I agree with FIRST in how they handled things. Well done.

Second, shame on whoever hacked the field. That is anything BUT what I was taught was proper FIRST behavior. What they did is unacceptable. As for their affiliated team, I think that whether they reveal themselves or not is their exclusive decision. If the team was truly not directly involved the incident, then I could honestly care less about who the team happens to be.

Third, After reading the individual team sections of the report, I can now say that this proves the not even the best teams are perfect and ANYBODY can can make a mistake. Lesson: every team should double check their work.

No further comment.

Steven Donow

14-07-2012, 14:38

For everyone who is saying that the team should come forwards:

Do you REALLY want that team's members to be known as "the team that ruined Einstein 2012" for years down the road? In their careers, even?

shortened for shortening reasons...

Ugggh. I can't agree any more with this post. Regardless of how much "gracious professionalism" people have, regardless of how kindhearted people in FIRST are, THIS IS WHAT WILL HAPPEN. Call me cynical, but this is just how people are.

The only circumstances under which a public apology should be posted are, in my opinion:

1) Completely anonymous, sent by individual to FIRST, then posted by FIRST.

2) Sent to the main contacts of the "attacked" teams, then posted by someone from one of those teams, anonymously.

3) Basically, anything that keeps the identity of the "attacker" completely anonymous.

Again, I would love to believe that the general FIRST community would be accepting of it, but no matter what, the team would be referred to as "the team that attacked Einstein"

techhelpbb

14-07-2012, 14:38

I don't think you understand what the actual problem was. The system is indeed vulnerable to a deauthentication flood, or even a fast trickle. However, there were no such attempts detected, and there is no evidence to suggest that any occurred. The testing did show that it was possible to disrupt the connection without triggering a warning, so the detection parameters need to be tweaked to something more appropriate to the FRC use case. Still, it doesn't look like this was something that actually happened during competition.

The confirmed problem was instead an unknown and unexpected bug in the access point firmware that broke the existing connection when another client tried to authenticate and failed. Nothing special needs to be downloaded in order to cause this bug to be expressed.

Unless you have something more than what is in the presented information I think you assume that AirTight which we now know can not see this issue under the right circumstances is sufficient reason to assume it did not happen. I draw nothing from this report that indicates to me they have actual raw data to confirm that a deauth attack did not happen on Einstein or elsewhere and it clearly is a well known vector with tools that often allow setup to exploit that hole in AirTight. Also you can make this work even if AirTight can detect it when fixed.

The second problem. The one you have listed as confirmed may be far more practical to point at and say well they did it and it requires no special tools we confirmed it. I actually mentioned that aggressive attempts to gain WiFi access could lead to this pages back. So I have acknowledged it but I think that concern about this and finger pointing is sort of crazy. There's actually premade devices you can buy that will locate and crack WPA passwords. Anyone could have brought one and had it in their pocket. Anyone at any point in the competition could have tripped over this. In point of fact well before Einstein suggestions about the versions of AP hardware surfaced. There where options to deal with this including spare parts in dumb luck you may get another version. The assumption here is that this person targeted teams with a mind to carry out rigging. How could they be sure the teams in question would be effected? They certainly can't walk over and offer to swap the AP.

On the other hand having now personally setup and torn down a field twice and looked at how AirTight is used. I am positive that a person could easily disable a robot or robots at will. They will still be able to do that when AirTight is patched and the AP versions are upgraded. I view this confirmation as a way to distract from the larger issue.

Racer26

14-07-2012, 14:49

Jay O'Donnell

14-07-2012, 14:51

First of all I'd like to say that FIRST has done a remarkable job in handling this problem. I can't think of another organization who could pull together this kind of report and would do as much work as they could to figure out the conclusions that they did. Bravo!

As for the interference, the individual in question got what they deserved and that's that. Whatever team they may have been a part of has no responsibility for them if they were acting by themselves. I personally believe that the team should stay anonymous because whether we like it or not, they will become "the team who ruined Einstein". Which of course they shouldn't, but unfortunately, that's just how it works.

As for the teams that were interfered with on Einstein, I hope all of you make a soon return to Einstein, because what happened is extremely unfair and selfish. For those of you that were effected by it, the FIRST community is here for you. All of these teams have shown the utmost professionalism, I don't know if there are many other teams thy could do so. While 180, 25, and 16 were a fantastic alliance and definitely have all the credentials of a championship alliance, I still feel sorry for certain teams who were unable to perform because of an outside attacker that they had no control over.

I have no doubt that us as a FIRST community will be able to get through this rough situation, and that the teams involved remain as successful as possible.

My $0.02

BrendanB

14-07-2012, 14:57

....

....

I can't agree with these posts more. At first I was stunned (still am), I think what happened is repulsive, I think an apology is owed, but it shouldn't be public. It should be anonymous to the 12 teams involved.

Some say that the team should step forward before it comes out later on. What if the team doesn't know a member on their team committed this act? Imagine if this was your team. Yes an apology is owed but if identities are made public that team and person will go down in FIRST history and not in a good way.

Ether

14-07-2012, 15:05

The majority of teams were running the affected firmwares for the robot radios.

Just for the record: for the radio, it was a specific hardware revision that was vulnerable (Revision A), not the firmware.

Racer26

14-07-2012, 15:08

techhelpbb

14-07-2012, 15:21

@techhelpbb:

The majority of teams were running the affected firmwares for the robot radios. You wouldn't have to be sure. It would affect most teams, and its unlikely you would have ended up with a full alliance of teams that couldn't be targeted in this way.

Okay but the effected Cisco firmware was not present until week 4 per the report.
Also I personally have Team 25s A version AP they traded me in an off season event.
It had been acting strange on them so they swapped it at a venue before Championship.
It is on my workbench next to me right now.
I gave them a brand new unit in exchange at an off season event.

The thing is that this issue did not exist until week 4. Team 25 had a version B so we know that version was floating around.
Someone would have to have discovered this issue from week 4, assumed that no one would swap the radio like 25 did for the sake of it, and had to have a grudge against that one alliance....then decided to continuously interfere only to have some fortune that wiring and other issues work to their advantage?

Seems far fetched. More importantly Team 25 had issues with that A version before that venue. If the target is 25 why bother to do it this way?

Andrew Lawrence

14-07-2012, 15:27

The people that make up FIRST are some of the smartest people on the planet.

Just for clarification, I've met some real dumb people involved in FIRST. Involvement in an organization that celebrates Gracious Professionalism doesn't mean a person follows those principles, and it certainly doesn't make someone smart.

EricH

14-07-2012, 15:43

While I would love to agree with EricH et al, I don't.

The people that make up FIRST are some of the smartest people on the planet. We can read between lines, look for evidence, and investigate. Many people already know who is responsible, and many more already know what team they were associated with. Lots of people will do their own investigation into what went on.

I recognize that some people will forever associate the team with the interference, HOWEVER, I think that with time (and not much more time at that), the information will come out from another source, and that will make the lasting impression much worse than if it is admitted to and apologized for by the team. To those conducting their own investigation: Leave the pitchforks and torches at home, please. Ditto for the tar and feathers.

What happens if the team is released now versus later?

Now, while emotions are running high, the team would probably end up on every blacklist in FRC. That means that nobody will want to deal with them. The team sponsors get wind of it, and possibly cut the team's funding. The team potentially folds due to the combination of lack of funds and stigma associated with being on everybody's "Don't pick them or accept them" list.

But what if you let it go for a year? Two years? Even four years?

Now we've got a new issue to deal with. A new game (and hopefully a flawless run in terms of comm issues) has left last year behind, and we've focused on a rule we don't like again. A different rule. A different issue. The team in question has had time to get some turnover--and to, if they choose, release that they were the team in question. The Einstein Incident has passed into memory--a painful memory, but one that can be looked back on without as much emotional turmoil. Two years, and there is more turnover. More memory loss/burying. By four years, there's probably been a complete student turnover, and possibly a large mentor turnover. Then you quietly release that such-and-such a team was at the root of some of the Einstein Incident--and people wonder what you're talking about. They go back, and go "Oh. That."

Now, I'm not saying that the team won't escape any consequences--I know enough about blacklists to know that some actions will put teams on them for years and years. As some of the top teams know who the offending team is, or have a reasonable guess, that team probably will run up against them--but not with them, due to blacklist--in eliminations at some point. But being on the blacklist of top teams versus being on the blacklist of every team that has one--well, if I was in that situation, I'd rather have somebody willing to play on my alliance in the eliminations.

Racer26

14-07-2012, 15:49

The thing is that this issue did not exist until week 4. Team 25 had a version B so we know that version was floating around.
Someone would have to have discovered this issue by week 4, assumed that no one would swap the radio like 25 did for the sake of it, and had to have a grudge against that one alliance....then decided to continuously interfere only to have some fortune that wiring and other issues work to their advantage?

Seems far fetched. More importantly Team 25 had issues with that A version before that venue. If the target is 25 why bother to do it this way?

Why are you focussing on 25? As you mentioned, 25 had version B at Championship (and so did 16). They were protected (whether they realized it or not) from the FCA vulnerability because of it.

techhelpbb

14-07-2012, 15:53

Why are you focussing on 25? As you mentioned, 25 had version B at Championship (and so did 16). They were protected (whether they realized it or not) from the FCA vulnerability because of it.

Simply because Team 25 had a working A version router that kept having issues on fields prior to this. So they swapped it.

Why did their A version have issues like that before that then have them go away when the B version was added. Perhaps someone targeted them before that.

They told me that swapping that unit was all they had to do to fix their mysterious failures.

I specifically handed them a new AP out of my own pocket cost because I wanted to see if I could find something in there to account for the issue.

I did the same for another team as well.

Meredith Novak

14-07-2012, 15:56

I also suspect that the 12 teams (or at least their representatives present at the investigation weekend) probably know, and were asked not to talk.

Not true. I was there on Einstein with our alliance and at the testing. The only think I KNOW is that is was no one on my team - our phones were off. We remember the days when you would be tossed from an event if you were seen with a wireless device near the competition field.

Lil' Lavery

14-07-2012, 16:02

You're taking an incredible moral high road, and saying that this person feels sorry. Good for you for believing in people, but what if they don't?! Unfortunately I've seen plenty of attitudes that say "I'm GLAD Einstein screwed up, because FIRST sucks and they had it coming". Paraphrased from many emails/FB comments/tweets/what have you, but that's the sentiment.
And what if they don't? What's the proper course of action if they have no remorse?

Should we continue punishing them until they break down and say their wrong? They've already been blacklisted, now we should publicly out them? What if that doesn't work? Should we continue to "turn the screws" until we see them suffer?

As I recall (I've only been to CMP once), the number of people allowed fieldside at Einstein would be restricted to the drive teams of those teams participating on Einstein, plus some volunteers.

That leaves a VERY small number indeed.
While I haven't been to St. Louis, that's simply wasn't true in Atlanta. I was field side on Einstein myself in 2009, despite not being a volunteer or participant on an Einstein alliance. There are numerous people who find avenues to be field side at Championship.

Also, 1717 in this video (http://www.youtube.com/watch?v=Vp6EQYAg1BI). Symptoms match again.
According to this post (http://www.chiefdelphi.com/forums/showpost.php?p=1165267&postcount=10), 1717's issues were caused by a bad cable on the robot.

Why are you focussing on 25? As you mentioned, 25 had version B at Championship (and so did 16). They were protected (whether they realized it or not) from the FCA vulnerability because of it.
Because he's on team 11 from Mt. Olive, New Jersey located about an hour from the home of Team 25. He's had personal interactions with team 25 on the issue at hand and knew of their usage of that firmware.

Steven Donow

14-07-2012, 16:06

Not true. I was there on Einstein with our alliance and at the testing. The only think I KNOW is that is was no one on my team - our phones were off. We remember the days when you would be tossed from an event if you were seen with a wireless device near the competition field.
Also, your team (as well as 25 and 180) were specifically cleared of guiltiness :P

Now, I'm not saying that the team won't escape any consequences--I know enough about blacklists to know that some actions will put teams on them for years and years. As some of the top teams know who the offending team is, or have a reasonable guess, that team probably will run up against them--but not with them, due to blacklist--in eliminations at some point. But being on the blacklist of top teams versus being on the blacklist of every team that has one--well, if I was in that situation, I'd rather have somebody willing to play on my alliance in the eliminations.

This. It's a shame for that team, and I'm sure the teams involved that have their reasonable guesses have harsh feelings towards that team, but there is absolutely no need for all of FIRST to be against that team as well.

Richard Wallace

14-07-2012, 16:08

...our phones were off. We remember the days when you would be tossed from an event if you were seen with a wireless device near the competition field.I remember those days, too.

I was a field volunteer at CMP (Archimedes) this year, but my crew was not selected to work the field on Einstein. While walking toward the stands before the finals, I passed the field volunteer crew that had been selected -- they were getting a briefing from their VC, and one of the items she emphasized was "turn off the Wifi on your phone."

jakemochas

14-07-2012, 16:46

Also, 1717 in this video (http://www.youtube.com/watch?v=Vp6EQYAg1BI). Symptoms match again.

According to this post (http://www.chiefdelphi.com/forums/showpost.php?p=1165267&postcount=10), 1717's issues were caused by a bad cable on the robot.

Several weeks ago we posted this response
We have not been able to determine why our robot died in any of our matches. Our Robot died on Friday in Qualification Match 101 on red 1. On Saturday, our robot died in Quarter-Finals Matches 1 and 3 on red 1. Finally, our robot died in Semi-Final Match 1 on Red 3. We died on Red 3 and Red 1 so our original thinking about it only having to do with red 1 was ultimately disproven.

We have thoroughly investigated our communication failures to the best of our abilities and at this time we still do not know the cause of the failures.

jakemochas

14-07-2012, 17:02

Also, 1717 in this video. Symptoms match again.

According to this post, 1717's issues were caused by a bad cable on the robot.

Several weeks ago we posted this response:

We have not been able to determine why our robot died in any of our matches. Our Robot died on Friday in Qualification Match 101 on red 1. On Saturday, our robot died in Quarter-Finals Matches 1 and 3 on red 1. Finally, our robot died in Semi-Final Match 1 on Red 3. We died on Red 3 and Red 1 so our original thinking about it only having to do with red 1 was ultimately disproven.

We have thoroughly investigated our communication failures to the best of our abilities and at this time we still do not know the cause of the failures.

Barry Bonzack

14-07-2012, 17:07

I do not know the individual involved, or what his/her motives were. If I were to meet the person, I would say this.

I'm sure you are aware of the hurt, shock, dismay, and costs that your actions have caused our community. In FIRST, we are taught to be gracious professionals. Grace (http://dictionary.reference.com/browse/grace?s=t&ld=1032) is defined as this:

mercy; clemency; pardon: He was saved by an act of grace from the governor. Synonyms: lenity, leniency, reprieve. Antonyms: harshness.

I'm sure you have volunteered your free time for the betterment of students in the past, and for that I thank you. I extend the benefit of doubt that whatever the motives, there is a possibility you meant to do good and chose an extremely incorrect course of action for doing so, which I hope you now see was a mistake. I forgive you of your actions, and wish you the best to all of your future endeavors separate from our organization.

Sincerely,
Barry B

Greg McKaskle

14-07-2012, 17:13

Since folks are attempting to identify FCA failures from videos and data from other events, let me share some diagnostic detail. In particular, a blinking RSL on a robot that isn't moving does not mean FCA.

If a robot radio never connects to the field at all, it is something else.
If it connects to the field but stops moving ...
Does the DS indicate that it is connected to the robot? The alliance wall light also indicates whether communication is successful.
If it shows battery voltage and other signs of communication, it is something else.
If there is no communication with the robot, it is time to determine if the radio and cRIO are on or off. The RSL doesn't convey much info, but if it is active, the cRIO is up and at least some of the user code is active.

That leaves us with the radio. The LEDs on the front show whether it is in AP or bridge mode and whether it is bridged.

The odd symptoms that point towards FCA are a robot that can be pinged even though no communications succeed and the robot will either return too quickly to be a radio reboot or will not return at all.

The reason the report mentions cRIO reboot times was to try and identify definitively whether other failures could fit the symptoms. Unlike the radio, this is dependent on the team's code and needs to be measured for each robot.

Greg McKaskle

torihoelscher

14-07-2012, 17:44

I think we should learn from this and make sure it doesnt happen again. FIRST has done a great job to handle this situation. FIRST asks us to be gracious and professional even to the people we dont like, if we do not act gracious then how are we any different from the individual? Everyone makes mistakes.

plnyyanks

14-07-2012, 17:46

I told myself I'd wait 24 hours to digest the report before posting.

First off, I think FIRST did an awesome job with this report, and totally exceeded my expectations. This is a great demonstration of the problem solving process, and it was handled extraordinarily well. Kudos.

As for the communication issues that arose, words can not express my dismay that an individual would intentionally sabotage matches like that. I think the rest of the thread has already appropriately discussed this, and I'll avoid beating the metaphorical dead horse.

With that being said, I think FIRST appropriately handled the situations they were situations very well, and I support their decision to keep the guilty party(ies) anonymous. Like EricH said (http://www.chiefdelphi.com/forums/showthread.php?&p=1177413#post1177413), revealing the team involved would cause a long-lived association between that team and the unfortunate events caused by one individual from that team. And that's not fair to the team - the majority of which weren't involved (and probably unaware). Sure, people will find out, but as a community, we should try and respect that team and not spread rumours or encourage "public hangings" of anybody.

I know that the FIRST community is composed of some of the best people, and I believe we'll get through it eventually. Let's try and keep cool heads and avoid deepening any wounds we've picked up.

JamesTerm

14-07-2012, 17:53

As I read the report, I've been looking for two specific points of interest:

1. Why was the there a significant amount of failures for just the red alliance teams?

2. Special interest in how much vision tracking contributes to network traffic, as we (beta team)... were concerned of overwhelming traffic from this for teams that wanted to process vision via driver station.

I believe point 2 with the network capping addresses this fear for future games... but I would like pursue this with actual numbers (not here in CD obviously). In short I'd want to know if the mjpeg frames cause a significant load in the network traffic, as I want to consider using mp4 compression going forward. I think FIRST action items will address some of this... I'm looking forward to getting these numbers!

As for point 1... this is something I observed while watching the Archimedes elimination matches, and the report also reflects this for the Einstein matches. I've observed that red alliances usually have won (e.g. 90%) this year in our 2 regionals (I wonder how true that is for other regionals). There may be some correlation of this and the higher seed being on the red team. I noticed for Archimedes elimination, an overwhelming victory from the blue alliances.

I don't want to talk about the political stuff but just want to throw some general things out there... 1. Is there a scapegoat political agenda going on? 2. No one can keep a secret as all will be revealed to those who want to know about it... it is just a matter of time. Why? friends tell friends, and those friends tell friends... and well you get the picture... just like FB itself.

(P.S. I do not want to know... I'm an engineer not a politician)

JosephC

14-07-2012, 17:57

Video here (http://www.youtube.com/watch?v=IMPc8myQAn8&feature=relmfu)

RSL is clearly flashing while it's dead, suggesting the cRIO is still running. Symptoms match what the report says happens with FCA.

This may have been caused by our Ethernet cable that connects to the radio loosening to the point that it was no longer transmitting data to the cRIO.

After Finals Match 1 on Newton we took the robot out to be examined. we noticed that the Ethernet cable was slightly loose, so we re-zip tied it to the radio. After that we ran fine in Finals Match 2. I'm not saying that this was 100% the reason why we lost control, but it does appear to be so, at least to me.

I'm leaving this here as a reminder to all teams, rookies to veterans, to double check EVERYTHING before going onto the field, you never know what could go wrong.

OliviaG

14-07-2012, 19:24

Chris Fultz

14-07-2012, 19:49

... 1. Is there a scapegoat political agenda going on?

What do you mean by this question?

Dad1279

14-07-2012, 20:21

As I read the report, I've been looking for two specific points of interest:

1. Why was the there a significant amount of failures for just the red alliance teams?
.....
As for point 1... this is something I observed while watching the Archimedes elimination matches, and the report also reflects this for the Einstein matches. I've observed that red alliances usually have won (e.g. 90%) this year in our 2 regionals .........

I just counted 71 red wins out of 150 matches for Archimedes... 47% red, 53% blue.

47R/95 for Alamo = 49.5% Red wins
31R/76 for Dallas = 40.8% Red wins

Disclaimer: I only counted once, so I may be off by a percent or two. Perhaps someone can crunch the numbers for all the regionals?

qzrrbz

14-07-2012, 20:33

I just counted 71 red wins out of 150 matches for Archimedes... 47% red, 53% blue.

47R/95 for Alamo = 49.5% Red wins
31R/76 for Dallas = 40.8% Red wins

Disclaimer: I only counted once, so I may be off by a percent or two. Perhaps someone can crunch the numbers for all the regionals?

I have to think he's talking about elimination matches, where there *should* be a decided red bias.

Racer26

14-07-2012, 20:39

Yup. There should be a fairly strong Red bias in Elims, especially so in the QFs.

Side note of interest, at ON2 (GTR-West) this year, 2056 played their first Elimination matches at a regional on the blue alliance. Every other regional, they've won from the #1 seeded alliance, who never plays with blue bumpers.

ouellet348

14-07-2012, 21:00

To those that believe that the team of the INDIVIDUAL should hesitate to come forward, let's think about our culture for a minute.

In my experience, FIRST is about accountability and responsibility. We have an accountability to others, both for our good decisions and our bad. We have a responsibility to make the world a better place than when we arrived in it. This means we hold ourselves accountable for our mistakes and celebrate our successes.

On my team I've made mistakes. We all do. We don't hide them from others. We learn from them. We discuss them. We are accountable to ourselves but also those within our community. I know for a fact at certain points during my time on the team many mentors would have called me an arrogant jerk, and they'd be right. But the culture of our team and of FIRST allowed me to accept my mistakes and grow. Now those same mentors have hired me at their companies and have given me great recommendations because I made myself accountable for my actions and learned from them.

Now about this individual. I do believe that they are beyond being forgiven. Their actions have a malicious intent that I believe is clearly illustrated by the actions of FIRST in dealing with them. FIRST doesn't reserve that kind of action for an accident. The individual's actions were deeply hurtful to everyone who has even been touched by FIRST, and I believe that they don't really care given,

1.Their actions
2. A lack of an apology thus far

That said they will only be anonymous for so long. Eventually the community will have a name and a number to go with it.

My opinion is that team should come forward. Help the community heal and acknowledge that a team member turned out to be a different person than you thought. Don't help protect this person for your own anonymity. That can only last so long. If you truly did nothing wrong no one who matters and has half a brain would hold you accountable for something that you are not to blame for. Don't protect someone who chose to make a decision that terrible. Address the situation in a manner befitting the quality of the community we are all a part of.

The teams targeted are some of the best. Ever. They have worked to excel to a point that most of us will never reach. They should be celebrated for reaching that level and helping inspire others to do so. I honestly think they deserve commendation for simply maintaining a positive attitude despite the malicious actions of some idiots over the years (and yes they are idiots, no apologies). They deserve an apology and an explanation. Anyone in their place does.

In short, to everyone. Be accountable. Acknowledge success and mistakes, and grow from them.

Just my thoughts on a bad situation that had a fortune to be dealt with by some amazing people.

Ether

14-07-2012, 21:07

Perhaps someone can crunch the numbers for all the regionals?

Attached Excel2000 spreadsheet is from the FRC Twitter data feed.

The usual caveats about the FRC Twitter data feed apply.

Ekcrbe

14-07-2012, 22:01

...
Now, while emotions are running high, the team would probably end up on every blacklist in FRC. That means that nobody will want to deal with them. The team sponsors get wind of it, and possibly cut the team's funding. The team potentially folds due to the combination of lack of funds and stigma associated with being on everybody's "Don't pick them or accept them" list.

But what if you let it go for a year? Two years? Even four years?

...

Now, I'm not saying that the team won't escape any consequences--I know enough about blacklists to know that some actions will put teams on them for years and years. As some of the top teams know who the offending team is, or have a reasonable guess, that team probably will run up against them--but not with them, due to blacklist--in eliminations at some point. But being on the blacklist of top teams versus being on the blacklist of every team that has one--well, if I was in that situation, I'd rather have somebody willing to play on my alliance in the eliminations.

Each team affected by this has a right to be angry at the person who perpetrated the situation. But if four years later the team comes forward or is exposed, and those teams hold a grudge and still can't forgive the team the individual was connected to, then those teams have a GP issue. I would hope that stigma wouldn't last too long that far down the road.

Everything else is spot on.

JamesTerm

14-07-2012, 22:16

Yup. There should be a fairly strong Red bias in Elims, especially so in the QFs.

Cool you and qzrrbz know what I'm getting at... I noticed a strong Red bias in my observations of regional elimination matches (Dad1279, qzrrbz is right I was talking about eliminations only) and I believe it to due to how the higher seed is on the red alliance. Now then when I observed the outcome of Archimedes eliminations... I noticed the blue wins as the #2 seed (blue) wins the division. Yeah... may not be significant, but noticeable to me.

Ether: Thanks for this data... it confirms what 1075guy and I have observed in regards to the red bias in elims. From this global scope xls document everything is probably as we'd expect. However, the FRC report itself does show a significant amount of failures for red alliance teams. Also, there was one match in particular that we played where we observed 2 simultaneous red alliance failures that got me thinking about this during the rest of the competition.

JamesTerm

14-07-2012, 22:34

What do you mean by this question?

I was hoping for an answer or opinion, not another question. ;)
It should be self-explanitory... if not then please just disregard.

Libby K

14-07-2012, 23:16

That said they will only be anonymous for so long. Eventually the community will have a name and a number to go with it.

My opinion is that team should come forward. Help the community heal and acknowledge that a team member turned out to be a different person than you thought. Don't help protect this person for your own anonymity. That can only last so long. If you truly did nothing wrong no one who matters and has half a brain would hold you accountable for something that you are not to blame for. Don't protect someone who chose to make a decision that terrible. Address the situation in a manner befitting the quality of the community we are all a part of.

This is what I mean. The situation in which the team formerly associated with the individual steps forward and makes a statement will create a much better atmosphere for themselves than the situation where the community eventually finds out who this person is and publishes their name and association. As much as I think that illustrates the bad parts of the internet, it's more than likely going to happen because this IS such an emotionally charged issue for so many.

If you were that team, wouldn't you want to admit that, like Alex said, this person had a different character than you thought... rather than get called out on a public forum before you get your chance to speak for yourself?

I don't approve of a witch-hunt for the person involved, but I do believe the team should get a chance to step forward before those with the torches and pitchforks get them first.

Steven Donow

14-07-2012, 23:39

This is what I mean. The situation in which the team formerly associated with the individual steps forward and makes a statement will create a much better atmosphere for themselves than the situation where the community eventually finds out who this person is and publishes their name and association. As much as I think that illustrates the bad parts of the internet, it's more than likely going to happen because this IS such an emotionally charged issue for so many.

If you were that team, wouldn't you want to admit that, like Alex said, this person had a different character than you thought... rather than get called out on a public forum before you get your chance to speak for yourself?

I don't approve of a witch-hunt for the person involved, but I do believe the team should get a chance to step forward before those with the torches and pitchforks get them first.

I disagree...I don't see how a team's statement would be any different announcing it themselves or after the "witch-hunt" finishes. I, for one, would feel no different feelings for the team other than, if they announced it themselves, saying, 'that's bold', and then moving on with the same 'not the team's fault, etc...' feelings. But as EricH greatly said earlier, that team will forever have their name associated with this, whether that's the right thing or not; it's what will happen.

The longer the secret is kept, the better. If the team name never is revealed other than mumblings at regionals(ie. "Yeah I hear it was that mentor for team XXXX"), then that is even better.

Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons? Part of me wants to view that as a case, but part of me has reasons why that wouldn't work (how would FIRST get into contact with the individual other than through the main team contact? unless that person kept it a complete secret/between them and the perpetrator).

DominickC

14-07-2012, 23:42

I'm happy for the FMS Whitepaper.

Racer26

14-07-2012, 23:47

Now then when I observed the outcome of Archimedes eliminations... I noticed the blue wins as the #2 seed (blue) wins the division. Yeah... may not be significant, but noticeable to me.

This is simply an artifact of the very high level play at Championship, where the top several (in this case 2) alliances are capable of winning.

However, the FRC report itself does show a significant amount of failures for red alliance teams. Also, there was one match in particular that we played where we observed 2 simultaneous red alliance failures that got me thinking about this during the rest of the competition.

The report also says that it believes the bulk of the communications issues experienced during Einstein (other than 118's code/custom electronics issue) are believed to be the result of the Failed Client Authentication exploit. Einstein shouldn't show the red alliance bias that other elimination rounds do. If the targeted teams happened to be red, that's not statistically significant.

Also, in answer to Greg McKaskle's earlier post about some robots DITW with a flashing RSL not being the result of FCA: Here's a likely example. GTRWest (ON2)'s Finals match 2 (http://www.youtube.com/watch?v=gSSnBhGY6R8&feature=relmfu), 1114 dies partway through, with a flashing RSL. It was a Wk5 event. The vulnerability existed then. Probably one of the non-FCA reasons.

The first shot of them dead on camera is when the match clock reads 87 seconds remaining. Immediately before this shot, the red alliance station is shown, their red light lit solid. Match clock 54, the red alliance station is shown again, solid light. Based on my understanding of the failure mode of FCA, a solid alliance wall light is proof of something else being at fault. At match clock 51 of Einstein SF2.2, 2056's alliance wall light is seen flashing.

Knowing how the FCA bug operates, its easy to see how it could be caused accidentally, and I wonder just what percentage of the post-Wk4 comms issues could be attributed to it, even if the person operating the failed client didn't intend or even realize the consequences of their actions (or the automatic actions of their Wifi devices).

Gregor

15-07-2012, 00:19

Each team affected by this has a right to be angry at the person who perpetrated the situation. But if four years later the team comes forward or is exposed, and those teams hold a grudge and still can't forgive the team the individual was connected to, then those teams have a GP issue. I would hope that stigma wouldn't last too long that far down the road.

Do they really have a GP issue? If someone killed another person and then waited 4 years to confess, would it be wrong for the family members to hold a grudge (an extreme case but you get my point)? As Tyler pointed out here (http://www.chiefdelphi.com/forums/showpost.php?p=1177396&postcount=165), no one but the twelve Einstein teams know what it is like to be cheated out of a fair shot at the gold. If I were them, I'm sure I would still be angry 4 years from now. Heck, I'm not them and I'll probably still be angry. This post gives a very personal view into what these teams had to go through this season. Why should they forgive this individual? I think there would be a problem if these teams were not still angry after some time has passed (say 4 years?). I have no problem with teams holding a grudge after putting in hundreds of hours, dollars, and heartache into their teams, and that is only for this season alone. Multiply by how many years each team member has been involved with their team, and you get the magic number of how angry each and every one of them should be should be.

Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons?

Thats what I read it as.
For personal reasons, this individual opted to resign.
Granted, "personal" reasons could mean anything from his/her dog dying to his/her overwhelming guilt for this inexcusable act of interference.

apalrd

15-07-2012, 00:20

Knowing how the FCA bug operates, its easy to see how it could be caused accidentally, and I wonder just what percentage of the post-Wk4 comms issues could be attributed to it, even if the person operating the failed client didn't intend or even realize the consequences of their actions (or the automatic actions of their Wifi devices).

I wonder how many times the cause was the team's own laptop (e.g. a programming laptop left on the robot cart). I'm currently very glad we don't use 802.11 directly to our laptops at home (we still use that Linksys AP from the 2009 kit), thinking about this.

Greg McKaskle

15-07-2012, 00:32

1. Why was the there a significant amount of failures for just the red alliance teams?
Let's look for a mechanism by which the alliance color could influence the failure.

Is it actually the color of the bumpers? Will putting red bumpers on a robot make it fail, or will blue bumpers fix it? Of course not.

What about the wifi? It turns out that all traffic is sent out on the same wifi frequency and channel. Red and Blue robot traffic is literally chopped into small packets and transmitted one after the other in a big data stream. RF spectral noise cannot bias red or blue or any particular robot. Additional APs using the same channel will not bias based on color or robot.

The data is sent out over the same antennae. The AP has six antennae, but three are tuned for 2.4 GHz and three for 5 GHz. The three antennae are used in MIMO fashion to modulate the bits of each packet. No bias there. The packets are the same size, and only difference is that the control packets have either an ASCII R or B within an inner field. B is 0x42, R is 0x52. The adjacent field has an ASCII 1, 2, or 3 for the alliance station by the way. Team ID is used as the SSID name. This resolves to a unique BSSID which is one of the six MAC address of the router. The MAC is in the IP packet header and nothing about team number, color, or anything else is. No opportunity for bias that I see.

That leaves us with the physical cables that deliver the data from the DS to the wifi AP and the switches that merge and route them. I don't have an explanation for how they would bias.

Finally, the report gives details as to root cause. I don't see failures following colors or stations. I mostly see things following robots.

Please explain the mechanism or let this one die. It is a small sample size and robots in elims do not swap bumpers.

2. Special interest in how much vision tracking contributes to network traffic, as we (beta team)... were concerned of overwhelming traffic from this for teams that wanted to process vision via driver station.

Vision processing that stays on the robot has no impact. Vision streams that are sent to the dashboard via TCP can be estimated and the utilization of the particular teams on Einstein was measured and compared to the estimate. A color 640x480 image stream has 921,600 bytes of pixel data per image. with compression, it is likely 30kB per image frame. If the team sets the framerate to the highest of 30fps, that gives less than a MB per second or about 10Mbits. For six robots that is a respectable 60Mbits for video alone. The rest of the field data takes about 6.5Mbits. True, some robots could have multiple cameras, and they could set compression such that images were three or four times larger, but most teams will not have 30fps and 640x480 and low compression. The Einstein fields were usually measured to use 25Mbits. I wouldn't want to use g networks, and n needs a clean channel to achieve lots of throughput, but the channel bandwidth is not taxed by vision cameras ... yet. mp4 compression will work fine if both camera and laptop have codecs. The cRIO doesn't.

I don't want to talk about the political stuff but just want to throw some general things out there... 1. Is there a scapegoat political agenda going on? 2. No one can keep a secret as all will be revealed to those who want to know about it... it is just a matter of time. Why? friends tell friends, and those friends tell friends... and well you get the picture... just like FB itself.
1. How about no. FIRST staff, experts, and the 12 Einstein teams in attendance were each asked if they approve of the report findings. All agreed. Explain why they would all have the same agenda?
2. Well if it is true for FaceBook, ...

Greg McKaskle

Ekcrbe

15-07-2012, 00:34

Do they really have a GP issue? If someone killed another person and then waited 4 years to confess, would it be wrong for the family members to hold a grudge (an extreme case but you get my point)? As Tyler pointed out here (http://www.chiefdelphi.com/forums/showpost.php?p=1177396&postcount=165), no one but the twelve Einstein teams know what it is like to be cheated out of a fair shot at the gold. If I were them, I'm sure I would still be angry 4 years from now. Heck, I'm not them and I'll probably still be angry. This post gives a very personal view into what these teams had to go through this season. Why should they forgive this individual? I think there would be a problem if these teams were not still angry after some time has passed (say 4 years?). I have no problem with teams holding a grudge after putting in hundreds of hours, dollars, and heartache into their teams, and that is only for this season alone. Multiply by how many years each team member has been involved with their team, and you get the magic number of how angry each and every one of them should be should be.

No, no, no. I'm sorry for any ambiguity in my previous post. I meant it would be wrong for the victims to be mad at the associated team, not the individual him/herself. After four years of time to numb the pain slightly, I would expect reasonable people, like FIRSTers, to have cleared their vision and then be able to see that the associated team itself is not (we hope) the bad guy. The rest of that team could be, in a roundabout way, like victims themselves, because of the horrible publicity potentially brought down upon them by the community at large.
I hope that hailstorm doesn't come to them.

To what you did say, of course they will be mad, and justified in being so.

Gregor

15-07-2012, 00:55

No, no, no. I'm sorry for any ambiguity in my previous post. I meant it would be wrong for the victims to be mad at the associated team, not the individual him/herself. After four years of time to numb the pain slightly, I would expect reasonable people, like FIRSTers, to have cleared their vision and then be able to see that the associated team itself is not (we hope) the bad guy. The rest of that team could be, in a roundabout way, like victims themselves, because of the horrible publicity potentially brought down upon them by the community at large.
I hope that hailstorm doesn't come to them.

To what you did say, of course they will be mad, and justified in being so.

When the individual's name and assoisted team are finally discovered, I would like to think that the FIRST community wouldn't hold the team to blame. I couldn't imagine it if someone from my team did something like this. That would already be agonizing for the team in itself. To then be know as the team who ruined Einstein 2012 will be a sad, yet albeit inevitable add on.

Steven Donow

15-07-2012, 01:09

what I read it as.

Granted, "personal" reasons could mean anything from his/her dog dying to his/her overwhelming guilt for this inexcusable act of interference.

Well, I was moreso saying that that's what they'd tell their team; "I'm leaving for random understandable personal reason" when it's really "I'm banned from FIRST events". With their team having NO knowledge whatsoever of what he/she did

And does anyone else think it might be beneficial(or in some ways detrimental) to have two topics of discussion for this:one political, one technical. Or would that just draw too much attention to the political stuff/become a battleground?

Ekcrbe

15-07-2012, 01:12

To then be know as the team who ruined Einstein 2012 will be a sad, yet albeit inevitable add on.

I challenge the FIRST community to be above this and avoid putting this stigma on an unfortunate team and consider everyone in this team's influence innocent until proven guilty. There's no guarantee the perpetrator acted alone, but it would be even sadder and stranger to find a cast of co-conspirators. I doubt this will be the case.

Akash Rastogi

15-07-2012, 01:53

Did anyone from 1717 get to formally investigate with FIRST the cause of their issues during division eliminations?

Seems logical to look into their case as well as other communication issues during eliminations at Champs, albeit, probably without inviting all teams to Manchester it might be difficult. At least it would give these other teams piece of mind as well knowing that it is not solely some issue with code/electronics that caused their issues during elims.

David Brinza

15-07-2012, 01:54

<snip> If the team name never is revealed other than mumblings at regionals (ie. "Yeah I hear it was that mentor for team XXXX"), then that is even better.

Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons?
I think we're all shocked by what occurred in St. Louis. This is NOT FIRST.

Suppose the perpetrator leaves the team because of graduation? The "individual" was not identified in the FIRST Einstein Investigation Report as being a mentor nor student. Wouldn't the FIRST community would be more forgiving if this was more like a teenage "stunt" rather than an attack on sportsmanship and GP by a team leader? If a mentor did this, the team involved likely would suffer irreparable damage to their reputation.

Todd

15-07-2012, 02:35

Good work on FIRST and all volunteers and afflicted team's parts with their swift and thorough analysis.

Good work on catching the issues with smart dashboard.

Good work on 118's part in identifying their own issue with their gyro reset loop.

Good work on 118's and volunteers parts again for working to identify the issue with their vxworks network buffer being overrun following the cessation of normal crio code activity.

Good work on all parts reminding us that its probably best that our cRios never run at 100% utilization, and that its perfectly plausible to not hit 100% doing all sorts of complicated logic, whether you're running c++, java, or labview.

Good work on deciding to limit team bandwidth moving into the future, our applications are only going to get greedier with time and we need to have a cap.

Good work spotting all wiring problems that were in fact present with several of the playoff teams.

And good work and god speed in all future efforts in securing the field against any external interference, be it intentional or not. The only secure network is one that's disconnected, but we can all (and I'm sure will all) do our best to improve the quality of experience for all FIRST teams.

JamesTerm

15-07-2012, 10:23

Let's look for a mechanism by which the alliance color could influence the failure.
...
Please explain the mechanism or let this one die. It is a small sample size and robots in elims do not swap bumpers.
Greg McKaskle

Thanks for the detailed analysis on this. I cannot explain the mechanism as I am not an expert in networking. I just observed one side having a signifcant amount of failures in the report. Since no-one acknowledged this in the report I wanted to... on the long shot chance that maybe (just maybe)... like tracing wires to a defected part... there could be something there.

The Einstein fields were usually measured to use 25Mbits. I wouldn't want to use g networks, and n needs a clean channel to achieve lots of throughput, but the channel bandwidth is not taxed by vision cameras ... yet. mp4 compression will work fine if both camera and laptop have codecs. The cRIO doesn't.
Greg McKaskle

Thanks very much for these numbers! Looking forward to seeing how they distrubute the network capping in future documents. One thing I'll want to determine is if the mp4 offerred by say the 1101, is if it actually uses p and b frames... or if it is i frame only. I know AVCHD like canon vixia does a great job with compression by using delta frames accordingly.

1. How about no. FIRST staff, experts, and the 12 Einstein teams in attendance were each asked if they approve of the report findings. All agreed. Explain why they would all have the same agenda?
Greg McKaskle

I don't think so either.

Al Skierkiewicz

15-07-2012, 10:44

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

IndySam

15-07-2012, 10:48

JamesTerm

15-07-2012, 13:08

There is one more thing I wish to share which was a symptom 100% reproducible where during the switch from autonomous to teleop showed no connection and we continued to have lost connection until near the end of the match.

This happened at our first match last year... Unfortunately I do not know the full reason as to why, but it had to do with how much work was done cleaning up objects that ran in autonomous and starting similar ones up again in teleop. The good news is that we could reproduce this problem using the practice button (So yes there is good reason to test with it). We had to disable autonomous for that competition to avoid this symptom.

I should add using the practice button is a great test, but it could give a false positive. It may be that the instantiation (or cleanup) is just on the threshold and with any more network time delay could throw it over the edge. I just want to throw a word of caution to all programmers to not make the same mistake we did. Be aware of how much work occurs between ending autonomous and starting teleop(). We never found the root cause of this, so I do not know if the vulnerability still exists.

What I can tell you is that we instantiate everything on powerup, and do the minimal amount of work to transition from autonomous to teleop.

In regards to the current subject at hand... I just want to add this as a diagnostic check if a similar symptom occurs or had occurred.

Foster

15-07-2012, 13:29

Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.
/end discussion
I've watched this for the last two days and have been trying to find a way to write what Al did.

Lets look to fix the control system problems. Remind roboteers and mentors that a 125LB robot moving at 9FPS slamming into another robot creates a lot of force so connectors, cables and COTS electronics may be damaged. Put as much design effort into our electrics and electronics as we do for our mechanics. Remind designers that CPU cycles and network capacity is not infinite, less is always more. Make every CPU cycle and network message count.

Lets pull together as a community on these topics. All teams now and in the future will benefit from the shared knowledge of how to do these things.

But as Al and IndySam said:

Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

Add me to that list too. Nobody benefits from retribution.

Taylor

15-07-2012, 13:48

I would like to know the team simply because it seems they could use a hug.

techhelpbb

15-07-2012, 13:59

There are ways to improve the time to troubleshoot power issues on the robot and they should be explored. Documentation is a fine thing but at some point we have lots of documentation now and all too often it's missed. Writing documents is fine to offload responsibility for risk to those that should be reading those documents. However, as Einstein demonstrates shifting responsibility does little for the reputation of FIRST generally. The people in the stands are there to see the show and robots not moving is what you expect at a museum.

There are proper ways and improper ways to conduct investigations if you know you have a real issue. I should hope the message of this report is tempered to consider that not everyone who takes issues with the status quo is going to operate with reckless disregard for policy and process. I should also hope that this report makes it clear to FIRST that stonewalling channels can lead to negative outcomes. Just a point as I have often found that it's a serious problem to get the attention of those that have the power to command resources to investigate within the organization. That alone can never justify the risk of this situation but I can see how the factor contributes.

From my last post it's clear that ranking or de-ranking Team 25 (based only on the unique information I have as I have their original A version router) couldn't have been the practical goal. Under the circumstances any effect on who won was probably as stated by Al and this report merely secondary to the person's goal to demonstrate the cause and effect of their ability to deny service to robots (regardless of whom was effected which in many ways made their point useless anyway).

Given the only person who has been confirmed to cause any issue was only doing something that anyone could have and probably did stumble over (hence some of the earlier failures). I return to the same point as the others. This was a bad decision on their part and they have paid their price according to what FIRST deemed acceptable. FIRST is the most effected. FIRST lost money, reputation, resources, and opportunity on a national stage as the direct result.

As to the idea presented elsewhere that no team can understand how these teams felt. Let me point out that frequently teams ranking towards the top are effected by the ruling of the refs on the behavior of other teams on the fields during matches. I have personally been involved with a team that felt they were improperly impacted by the actions of other teams and people on the field during play. While this is more unusual than normal it's really quite similar. These are all fantastic teams. I feel badly for the seniors on these teams who will soon no longer be able to hold student roles, but beyond that these teams are not magically great. They have important recognized qualities that will carry them past this and we hope to future recognition.

Let no one mistake that I still think the smart thing for FIRST to do is legally bind these people to silence and strongly discourage any attempt to discover who these people are. We are a large organization. I worry not about the majority of FIRST and how it will behave but I worry that somewhere in FIRST there is a small number of people that will play hero if they find out and only compound the misdeed.

Let us focus on the positive which is that FIRST can move forward into the future hopefully better for this.

Lil' Lavery

15-07-2012, 14:30

Is it actually the color of the bumpers? Will putting red bumpers on a robot make it fail, or will blue bumpers fix it? Of course not.

Well no, but it could influence the drivers.
http://www.sciencentral.com/video/2008/08/19/wearing-red-an-olympic-advantage/

:cool:
/tangent

akoscielski3

15-07-2012, 16:42

Well no, but it could influence the drivers.
http://www.sciencentral.com/video/2008/08/19/wearing-red-an-olympic-advantage/

:cool:
/tangent

This is probably why i am more scared of 1114 than 2056. lol (I love them both though :) )

BrendanB

15-07-2012, 18:16

This is probably why i am more scared of 1114 than 2056. lol (I love them both though :) )

I wonder what is going through their minds when they face each other! :P

Ekcrbe

15-07-2012, 18:21

I wonder what is going through their minds when they face each other! :P

Especially if 1114 is in blue and 2056 is in red...

Matthew.Mc

15-07-2012, 18:41

Especially if 1114 is in blue and 2056 is in red...

I believe he's referring to the colour of the teams individually, and not when there on the field (Bumpers).

Ekcrbe

15-07-2012, 18:42

I believe he's referring to the colour of the teams individually, and not when there on the field (Bumpers).

That's the point! If their team colors oppose their bumper colors, who has the advantage?

Jessica Boucher

15-07-2012, 18:52

I'm really proud of FIRST and its volunteers for putting such a thorough report together. This was a fascinating thread to read, and I appreciate everyone pretty much holding it together and providing a good discussion.

Although unrelated to the report, it did make me finish my recap of Championship. You can read it here (http://tresboucher.com/2012/07/15/memory-is-deceptive/).

Gregor

15-07-2012, 18:57

That's the point! If their team colors oppose their bumper colors, who has the advantage?

Everyone else :D

akoscielski3

15-07-2012, 19:19

That's the point! If their team colors oppose their bumper colors, who has the advantage?

DUNT DUNT DUHHHHHHH

lmfao

Joe Johnson

15-07-2012, 21:23

With the passage of time, things get clearer in my head but I don't know if I am happier about the situation.

First, for my own part, I apologize unreservedly to FIRST Canada. My early post was totally unfounded. I am sorry for implying that perhaps this was in any way connected to bad blood associated with the Greater Toronto East Regional. This was very unfair of me.

Second, as to naming the individual and the team, it is clear from the various eye witness accounts that a LOT of people know who this person was and what team they were associated with. KEEPING THIS A SECRET IS UNTENABLE. It will get out. Once it does, I suppose that there will be a lot of digging around looking for reasons why this information was kept secret... ...and I assure you that it is always possible for conspiracy theorists to weave together plausible narratives that will paint the individual, team and FIRST in a much worse light than if they had published the information themselves.

I BEG FIRST, the associated team and even the individual to publish what they know. While it is painful in the short run, it is going to be better to get this out in the light of day now.

While they are at it, they should probably address the rapid departure of Bill Miller. Even the most charitable observer cannot help but wonder if his departure was related to all of this. I am not saying it is or it isn't but I think it is in FIRST's best interested to address this openly.

Calling 'em as I see 'em.

Joe J.

Mr_I

15-07-2012, 21:28

I would like to know the team simply because it seems they could use a hug.

The best thing about forum discussions is that we don't need to be physically with the person (/ team / entity) to give them a cyber-hug.

So who's with Taylor and me, and wants to give the currently (and hopefully permanently) unknown team a group hug?? Come on, CD community, let's set an internet record!

Cyber-Group-Hug time! :)

CalTran

15-07-2012, 21:35

The best thing about forum discussions is that we don't need to be physically with the person (/ team / entity) to give them a cyber-hug.

So who's with Taylor and me, and wants to give the currently (and hopefully permanently) unknown team a group hug?? Come on, CD community, let's set an internet record!

Cyber-Group-Hug time! :)

Does that mean we're having another unofficial ChiefDelphi web hug?? :D :rolleyes:

plnyyanks

15-07-2012, 21:38

Does that mean we're having another unofficial ChiefDelphi web hug?? :D :rolleyes:

Web-ier and hug-ier than ever before! But wait! There's more! Call within the next 10 minutes and we'll throw in an extra hug FREE! (just pay processing and handling)

OliviaG

15-07-2012, 21:46

Web-ier and hug-ier than ever before! But wait! There's more! Call within the next 10 minutes and we'll throw in an extra hug FREE! (just pay processing and handling)

Wow a web hug??

DonRotolo

15-07-2012, 22:42

Virtual, of course. :)

I've waited until the report digested properly to comment.

The short version is: Good job everyone, that's a good report. Thank you. 1676 suffered lots of comm problems throughout the season, and I'm 98% convinced that our code is at fault.

Now for the hard part: Finding it...

Greg Needel

15-07-2012, 22:57

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

I was standing by the field during Einstein with no team in the competition, only for the opportunity to be there with friends from other teams and try to give them support in the finals. I can honestly say I have never had such a range of emotions brought on by any series of events then these finals. I witnessed the discussions, I vented my frustrations, and I offered my sympathies and congratulations to my friends. One thing remains is that I have never been so angry at a robotics event, to the point that I felt physically ill after the event. ALL THIS WITHOUT MY TEAM COMPETING. My heart will always go out to my friends on the teams that will have to live with the feelings of being taken advantage of, and the thoughts of "what if" for the rest of their lives. Sure, the pain will lessen but you never forget these things.

Afterwards many people, including myself, second guessed how FIRST handled the situation and wished that things had been done differently regarding on the spot trouble shooting, but not having been in their shoes and knowing that hindsight is always 20/20, I have nothing to say on this aspect of the situation.

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure. In all honesty, part of me wishes that it had just been a random occurrence caused by almost anything else. The fact that someone knowingly sabotaged the matches is UNFATHOMABLE in the community which we have. Not only has this person impacted the lives of the teams he stole from, but has brought into question how trusting we can be with our community. I hope that this is an isolated incident, but sabotage or interference will now be in the minds of every competing team for both future and past events.

The one thing that is left still unresolved in this whole situation is a public statement from the individual and team involved.

The person who did this needs to stand up for his actions, not just to provide a sense of closure to the situation, but also to grow from his mistakes. One of the hardest things in life a person has to do is admit that they were wrong, and while the punishment has already been dealt out, it is a showing of personal maturity about the way the situation should be handled. When you make a mistake you must come clean for both the people impacted by your decisions, and to clear your conscious. In addition to a public apology I believe the right thing to do would be to reveal each instance that this exploit was used in order to clear the air and remove the doubt that has clouded the minds of every competitor at events from week 4 onward.

It is going to be a rough road if (and hopefully when) you do come forward, but you should sleep in the bed you make, and in the end I promise you that you will be better person for it.

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.

Greg

AdamHeard

15-07-2012, 23:06

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability from your actions.

Greg

Greg, I really respect this post and your opinion.

Akash Rastogi

16-07-2012, 00:03

it is neither gracious or professional to not have personal accountability for your actions.

Greg

+1

This last part of your post really stood out to me. Accountability is also a way for you to earn at least a little respect back from those you have wronged; some sort of decency is kept. The speculation and rumors that spread amongst the community may also sometimes breed animosity for the entire team, regardless of who took part in the wrong-doing. The longer this team/person holds out on some form of apology, the less and less respect, forgiveness, and sense of decency they should expect from the FIRST community and especially the teams they affected.

On another note- just read tonight that a team who will be attending IRI was able to fix bugs in their robot thanks to the Einstein report and I hope this is the case for many others, so +1 to all those who took part in the testing.

AlexD744

16-07-2012, 00:30

Lil' Lavery

16-07-2012, 00:30

techhelpbb

16-07-2012, 08:27

"You understand we're making a conscious choice to do the boring version of the story?" (http://www.hbo.com/the-newsroom/index.html#/the-newsroom/episodes/1/03-the-112th-congress/video/clip-times-square-bomber.html/eNrjcmbOYM5nLtQsy0xJzXfMS8ypLMlMds7PK0mtKFHPz0mBCQ Ukpqf6JeamcjIyskknlpbkF+QkVtqWFJWmsjGyMQIAWCcXOA== )

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.

The idea that there's a person involved has captivated people's attention from the simple points that will still be unresolved regardless of any apology.

The best of the best had power issues that no amount of reading would find.

We have been distracted by the communications about this person from the larger security issue of deauth and other attacks which remain possible.

We've ignored the much more annoying aspect that some of these other attack vectors won't actually stop the robot and leave it stopped they can be used to manipulate the field play (stop a robot at just the right moment just long enough to effect match outcome and in the process make finding the source much more difficult).

It's additionally not clear to me that certain documentation regarding programming is sufficiently direct enough to communicate to teams what things they can do in a particular language that will cause issues but other than that seem perfectly reasonable.

There's so much more to this than just what this person did or other people probably stumbled on earlier in the season.

Ed Law

16-07-2012, 10:36

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

...

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.
Greg

I too waited a couple of days before I post. I read the letter, the report and every post on CD regarding this. I agree with Greg and others that the team and the individual should come out in the open and explain what happened. Let's not worry about other people's reaction when they find out. If they want to be angry, let them be angry. If they want to hold a grudge against the team forever, let them do that. If they want to forgive and move on, let them do that. We have no control on what other people think. It should not be a reason whether the individual and the team should come out in the open or not.

I am not making wild speculation here. Based on what I have read and putting my logic hat on, the individual seems to be one of the mentors of one of the Einstein teams. Based on the report and posts on CD, three of the four alliances can be cleared for various reasons. That leaves one alliance of 3 teams. Considering what happened in the semifinal round, there is motive why this individual may want to talk to the FTAs. This part is speculation so I am not going to name the team. I may be completely wrong here.

I think the team will come forward and do the right thing. I also don't think they should accept the free registration for next year. Let's give them some time. The report just came out over the weekend. They probably need to have a team meeting to discuss this and let everyone on the team know and draft a proper statement.