Team 548 Einstein Statement

[DMetalKong]

Originally Posted by Alan Anderson

An 802.11 protocol change that encrypts "management packets" could probably prevent deauthorization flood attacks from succeeding. It would also break a lot of things in the process.

I think that breaking things could be acceptable for use in FRC if the need is strong enough. As long as only firmware is changing, and not hardware, the cost of deployment would not be as great as an entirely custom solution.

Originally Posted by Alan Anderson

Did you read the Einstein investigation report through to the end? The last two pages are all about planned possible changes, with a half dozen of them as specific solutions to observed problems.

I did read the report I was referring more to the various threads on CD that have been started about the topic.

[stjonl]

Originally Posted by Gray Adams

I want to echo this point. By the mentor's own admission, he used the attack, but why should we believe his admission of guilt isn't the full story from his perspective? Every single one of us has been looking for someone or something to blame for what happened on Einstein. The full report has brought up a multitude of points of failure during the finals, and its really not hard to believe the answer to all of this is not as simple as blaming this all on one mentor. As soon as news broke that there was an attack during play, all of the failures on the field were attributed to that. But things just aren't that simple, and we discovered how many root causes for all the different problems there really were. But I firmly believe we still know far too little to place all of the blame on this one attacker. With thousands of incredibly smart people in the dome, its entirely possible that someone else used this attack, whether or not their team was on einstein, and whether or not they were fully aware of their actions.

We've heard 2 sides of the story so far, and unless someone would like to point out something I missed that puts them in direct conflict, I think it's only fair to evaluate this based on what we know.

Everyone was feeling a lot of emotions at the moment, and the attack in response could have been from a moment of desperation. I'm not condoning what happened, but I am trying to understand it.

I think part of the story happen before St. Loius. At MSC the finals were 469, 67 and 830 against 2054, 548 and 245. The red alliance won the first match, but the second match ended in a most unusual note. 2054, 548 and 245 were attempting a tripple balance. A little before that, 67 was in the blue alley and died about two feet in front of the blue bridge. The blue alliance charlie browned the bridge and contact was made with 67 a few times. At the end of the match, the blue bridge was level, but one robot, on the blue alliance side was half on the bridge and the floor. The referees looked it over and huddled up. I believe they call a few 3 point penaties during the match for contact in the alley. The final score was close enough that a blue bridge balance would have given them the match win and force a thrid final match. I can only assume the referees were discussing if there was a bridge balancing interference. As they discussed the issue, the winning teams were call to the floor to cut down the nets. Referees still discussing. Some of the nets have now been cut down. One referee leaves the huddle, removes his striped shirt within a couple steps and is clearly not happy. The rest of the nets are cut down. I can not remember exactly when the referee huddle ended, but at some point the MC them explains that the balanced blue bridge did not count because of the one robot half on the bridge and the floor. No mention of the possible bridge balancing interference. Myself and many others believe that the bridge balance was interfered with, OK, not deliberately, but just the same. That's not how the referees called it. 2054, 548 and 245 could have been denied a chance to win the tournament because of that call. Referees are human and they do the best they can, do not blame them unless you want to fill their shoes.
I was really suprised that the referee huddle was still in progress when they began the ending ceremonies. That really dampened the mode for everyone there.

[Siri]

Originally Posted by EricVanWyk

At an event, the "question box" is the best way to begin communication, you just need to be patient as your question gets routed to the best person to answer it. Outside an event, email is your best bet. Specific to these types of situations, you can use 2012frcfeedback@usfirst.org (as stated in the Einstein report). Please note that many people are currently on vacation, and the ones that aren't are buried in work…

While I agree with you, most significantly on your discussion of communication, the outside-event lack-of-communication issues described lately on CD are far from the first time I've heard intelligent, articulate and patient people report silence from FIRST. Without saying they're right (or wrong), I will contend that it's widespread enough to attract the attention of myself and others.

As for at-event, I've already put in my defense of the question box, but my experience somewhere between yourself and EricH. I've witnessed and experienced mentors communicating issues to students for the question box from both sides, and I'm not sure it's routinely as slow or unwieldy as some are concerned (though it certainly has the potential to be...as a coach I actually review question box procedure with my drivers). In this situation, I do believe officials would listen, especially after the second failure.

At the same time, there hasn't been a year go by (as driver, coach or volunteer) that I haven't seen at least 2-3+ patient, articulate, clear-but-not-obnoxious students, some with documentation or other teams to back them up, never be given the chance to talk and instead get sent away from the box. Nor is it altogether uncommon for these students to be independently proven correct, but only later than necessary and in some cases too late. (The obvious best-known example this year was at championship quals, though thankfully then it wasn't too late.) How many more have had a valid point, and how much difficulty on both sides did the breakdown in communication cost? I understand and feel very dearly for FTAs and my fellow refs, but I can't help but try for alternatives. As this situation demonstrates, there's something to be gained for everyone.

Am I making a mountain of a molehill? Maybe, but it's if so it's far from an uncommon hallucination.

[Adam Freeman]

Originally Posted by stjonl

I think part of the story happen before St. Loius. At MSC the finals were 469, 67 and 830 against 2054, 548 and 245. The red alliance won the first match, but the second match ended in a most unusual note. 2054, 548 and 245 were attempting a tripple balance. A little before that, 67 was in the blue alley and died about two feet in front of the blue bridge. The blue alliance charlie browned the bridge and contact was made with 67 a few times. At the end of the match, the blue bridge was level, but one robot, on the blue alliance side was half on the bridge and the floor. The referees looked it over and huddled up. I believe they call a few 3 point penaties during the match for contact in the alley. The final score was close enough that a blue bridge balance would have given them the match win and force a thrid final match. I can only assume the referees were discussing if there was a bridge balancing interference. As they discussed the issue, the winning teams were call to the floor to cut down the nets. Referees still discussing. Some of the nets have now been cut down. One referee leaves the huddle, removes his striped shirt within a couple steps and is clearly not happy. The rest of the nets are cut down. I can not remember exactly when the referee huddle ended, but at some point the MC them explains that the balanced blue bridge did not count because of the one robot half on the bridge and the floor. No mention of the possible bridge balancing interference. Myself and many others believe that the bridge balance was interfered with, OK, not deliberately, but just the same. That's not how the referees called it. 2054, 548 and 245 could have been denied a chance to win the tournament because of that call. Referees are human and they do the best they can, do not blame them unless you want to fill their shoes.
I was really suprised that the referee huddle was still in progress when they began the ending ceremonies. That really dampened the mode for everyone there.

I am not quite sure what your post has to do with this topic. It's not even correct. The field was given the "all clear" signal and the scores/winners were announced before the nets were cut. Not sure what was going in with the refs (maybe Gary Voshol can clarify). I know there were some upset mentors from the blue alliance. Heck, even the drive coaches for the winning alliance were less than enthusiastic with the way things ended.

[Al Skierkiewicz]

In case everyone doesn't know, I will state this again. Key volunteers are asked to report on their events following each of those events. We have weekly phone conferences to answer questions and pass along the latest data. Specifically, that is the FTA, LRI, and Head Ref as well as others. Those reports will contain information that is beneficial to each of the groups, to improve the next weeks events and FRC in general. In some cases, these reports will prompt a Team Update and/or a rule change. As an LRI I can assure you that one of the persons who hears my report is a member of the GDC and I have access via email and phone to First Engineering. If one of us comes across something that we can verify at the event, then we make the tests and insure that there is a real problem that exists. Then we all document it and report it through our individual lines of communication with HQ along with fixes that may have been found. Reports that come to HQ through other means are also evaluated and checked. While some people on this thread believe that they are not listened to or that they are ignored is simply and categorically untrue. They are opinions, not fact. While I can't really speak for the other volunteers, I can tell you that robot inspectors, FTAs, Refs and First staff are dedicated to improving this competition environment. Those outside of the robot key volunteer organization i.e. event staff, coordinators, pit admin and safety advisors are also dedicated to improving things. If you want to know who those people are, you only have to look here at CD and see who is answering certain questions in a helpful, open manner without trying to be condescending, boastful or argumentative. I can tell you from personal experience, trying to evaluate a problem with few staff and come up with a solution in a week or less is very difficult. That pushes confirmations, phone or email, way down on the list or priorities. Most of the First staff are at events throughout the season so that leaves even less time to come up with solutions. First staff and GDC are always looking at CD for input and they are reading exactly what you write here even if they don't respond.
As Alan pointed out earlier, the Einstein report hints that the 802.11 protocol allows for various types of security and those suggestions from the experts are being employed for next season. Other suggestions related to antenna designs, placement of components and other issues with the wifi infrastructure are also being implemented to insure secure communications with your robot.
David and Siri, let me know if this didn't answer your questions. Alan and Eric thanks for your input.

[qnetjoe]

Originally Posted by DMetalKong

I think that breaking things could be acceptable for use in FRC if the need is strong enough. As long as only firmware is changing, and not hardware, the cost of deployment would not be as great as an entirely custom solution..

There is really no sense in reinventing the wheel. IEEE has been working on protected management frames for a long time. There is a standard called 802.11w-2009 that does this and it was ratified in 2009 and was recently superseded by 802.11-2012 which was just the merging of ten amendments together to help prevent forking.

The next step in the process would be to find hardware that meets FIRST budget and that meets these standard.

just my two cents

If you want to track the 802.11 WG progress/projects here a good link to start off with

http://grouper.ieee.org/groups/802/1..._Timelines.htm

[Racer26]

I'm admittedly a bit late to the party on this one, but here goes anyway.

IMO, 548's statement is what was needed to get the community to put away the pitchforks. BUT, I don't believe the story told to them by the mentor in question. I personally feel that both the story told to 548's committee by the mentor, AND the story in the Einstein Report BOTH overlook the giant elephant in the room, that being the mysterious comms losses through MSC and Newton Elims, exhibiting symptoms consistent with the FCA attack (based on my viewing of the match video I can find), and only to teams that would pose a threat to 548's success in future matches (or current matches, in a desparation effort to avoid elimination).

I'm content to take 548's committee at face value, that this mentor acted alone, and without the knowledge of the rest of the team. However, I feel that the individual who interfered with Einstein isn't telling the whole truth. It is my opinion, and one I'm sure I'm not alone in, that the individual had been using the FCA vulnerability since at least MSC.

Since I believe this individual was acting alone, I hold no grudge against 548 as a team.

Another thing I haven't seen mentioned, is that there was a Week 4 event held in 548's school. Its entirely likely that the mentor discovered the vulnerability there, when they would have had decidedly greater access to a real field than anywhere else.

While I agree with many of the posters in the first couple of pages of this thread that interfering at the highest stage to demonstrate your vulnerability isn't cool, if this person hadn't done that, we probably never would have had the Einstein Investigation, and many of the issues uncovered by it may have continued to go unnoticed.

[Ekcrbe]

Originally Posted by 1075guy

I personally feel that both the story told to 548's committee by the mentor, AND the story in the Einstein Report BOTH overlook the giant elephant in the room, that being the mysterious comms losses through MSC and Newton Elims, exhibiting symptoms consistent with the FCA attack (based on my viewing of the match video I can find), and only to teams that would pose a threat to 548's success in future matches (or current matches, in a desparation effort to avoid elimination).

It's not really an overlook, because the Einstein Report reported on Einstein, and was not obligated to anything else. I believe (anybody confirm?) that FiM is investigating other potential instances of FCA attacks outside of Einstein.

Edit: It seems there is some investigation into matches at MSC by FiM members.

[techhelpbb]

Originally Posted by qnetjoe

There is really no sense in reinventing the wheel. IEEE has been working on protected management frames for a long time. There is a standard called 802.11w-2009 that does this and it was ratified in 2009 and was recently superseded by 802.11-2012 which was just the merging of ten amendments together to help prevent forking.

The next step in the process would be to find hardware that meets FIRST budget and that meets these standard.

just my two cents

If you want to track the 802.11 WG progress/projects here a good link to start off with

http://grouper.ieee.org/groups/802/1..._Timelines.htm

The Cisco 1250 & 1260 series AP (with 32MB) already have support for Cisco Management Frame Protection (MFP). It's a similar idea but prior to the full standard availability. It requires a specific configuration and it requires a Cisco Certified Extensions CCX version 5.0 compliant device. There is support for CCX v5 from Ralink, Atheros and Broadcom. Though mostly as peripherals not in routers, AP or bridges that I could find. So it saves the field side but not the robot side.

Rosewill sells an USB device claiming CCX v5 support but I can't vouch for that personally, the model is:
RNX-N600UBE

I'm not making any recommendations here, if anyone is really interested please discuss the matter with AirTight regarding any caveats.

[Liz Smith]

There is a big issue I see that has been echoed many times in these Einstein discussions. There are problems with assumptions that are not necessarily correct, speculation without any data, and research without the proper tools. I will use myself as an example, but I feel this applies to everyone.

It would be presumptuous for me to consider myself, for example, a FRC wifi interference researcher just by sitting here at home watching YouTube videos of matches. Any information I gain is anecdotal at best. I do not have a full set of data, I do not have a full FRC field in my house (yet??) to conduct my own research—and that’s not necessarily a bad thing.

But... lets say I’m really concerned about the second to last match in the “Regional State District Division Championship” where my team stopped moving during the match. It is way too easy for me to watch a video reply of that match 100 times and convince myself without a doubt that I know why that robot failed.

This is a problem. I can theorize all I want, but it is counter productive for me to arrive at absolute conclusions because I have neither a full set of data or the proper tools to investigate. It would be much more productive for me to report my information and concerns and maybe a link to the video, but then let go of it and let someone else with the tools to fully investigate the issue look into it. If I pursue it further, I just end up making more assumptions and run the risk of assuming that every robot that stops moving in every match I ever watch is because of this one reason… which in reality can’t possibly be true. Going further, if I then go out and present myself as an expert on these matters all I am doing is spreading rumors without any real empirical data to back it up.

I can definitely voice my concerns, but I have to accept the fact that the people whose full time job it is to solve these problems are going to be better equipped than me to draw these conclusions. I know that FRC staff, volunteers, mentors and students... we all want every single robot on the field to function properly 100% of the time but when theories are presented as factual evidence and people make statements that are just speculation it just causes unnecessary confusion.

[Basel A]

Originally Posted by 1075guy

I'm admittedly a bit late to the party on this one, but here goes anyway.

IMO, 548's statement is what was needed to get the community to put away the pitchforks. BUT, I don't believe the story told to them by the mentor in question. I personally feel that both the story told to 548's committee by the mentor, AND the story in the Einstein Report BOTH overlook the giant elephant in the room, that being the mysterious comms losses through MSC and Newton Elims, exhibiting symptoms consistent with the FCA attack (based on my viewing of the match video I can find), and only to teams that would pose a threat to 548's success in future matches (or current matches, in a desparation effort to avoid elimination).

Based on my understanding, it is impossible to reasonably diagnose a failure as FCA based on match video. Indeed, the symptoms consistent with FCA are consistent with roughly a million other problems. Even after FIRST's thorough investigation, all but one case of likely FCA cases were only considered "likely."

I find it disturbing that you're prepared not only to diagnose a robot failure as a complex problem based on minimal evidence, but also ready to indict an individual, about whom you know exactly one thing, of match-fixing at the highest level.

[Ekcrbe]

Originally Posted by Basel A

Even after FIRST's thorough investigation, all but one case of likely FCA cases were only considered "likely."

And that case (SF 2-1) was only confirmed because of the individual admitting to it, so there was no testing-based confirmation of FCA being an absolute conclusion.

[Gregor]

Originally Posted by Basel A

I find it disturbing that you're prepared not only to diagnose a robot failure as a complex problem based on minimal evidence, but also ready to indict an individual, about whom you know exactly one thing, of match-fixing at the highest level.

I don't believe that he is diagnosing them as FCA attacks, only pointing out the possibility of more FCA attacks that might have happened before Einstein.

[Greg McKaskle]

Diagnosis based solely on video is highly speculative. But knowing more about the behavior of any dashboard feed, the diagnostics on the DS, and any after-match troubleshooting results can move the needle as far as likely. It should require hard evidence or admission to move it to confirmed.

Greg McKaskle

[techhelpbb]

I'm trying to make this my last post in this topic...

This is a summary of the advice I've given my team:

Wireless networks like this are assumed to occasionally be unreliable and in order to handle the added complexity they implement solutions that may or may not be sufficient to make them as reliable as possible.

It is wonderful what the specifications for these networks would lead you to believe as a selling point for that technology. It is wonderful what demonstrations you can make to test those specifications in one circumstance or another.

However, at the core this technology creates a link subject to some unreliability even when you don't have someone trying to make it unreliable intentionally.

It's wonderful that FIRST is trying to make these links as reliable as possible but we as the robot builders can help by making our robots less dependent on the wireless network being entirely reliable for every instant we use it.

If the parts of the robot we as the robot builders control are less dependent on the reliability of the wireless network it will be much harder for an unforeseen situation over a short period of time to decrease our competitive performance. Regardless of whether that short interruption is from someone trying to cause trouble or an unforeseen circumstance.

Our team is student-led. I'll let them decide how to deal with that. I'm confident there are many things they can do with that advice to improve the competition performance of a robot.

This advice leaves FIRST additional room to have undetected problems in their network for short periods of time for a large number of possible reasons. So this is intended to be constructive and positive leaning guidance. I think it's fair to point it out because I don't expect that it's entirely level headed to charge FIRST with doing something quite hard with a great number of variables and expect there to be no issues along the way.