[FRC Blog] Einstein Report Released

[Billfred]

Originally Posted by ttldomination

Oh man, I have never seen someone roasted so hard on CD.

Just stop and think for one second. You know there's an issue, and you know the people who can fix it. But, instead of acknowledging the issue, they tell you to go away? Then, you're determined to show them that this issue is real, and that it matters. All you can think of is proving them wrong and proving yourself right. So then you take actions that aren't good, but in your mind, they will serve a greater good.

In my short few years being around people, I've met a handful of people who are utterly brilliant but they have no social awareness and a lack of ability to see consequences. These people are nailed as socially awkward, but in their mind whatever they are doing, however they are doing it, is perfectly right.

As much as I can, I use the concept of gracious professionalism as an internal yardstick. However, I have a really hard time believing anyone would think that the correct way of carrying yourself when in possession of an issue like this is to disrupt the climax of FIRST's largest, most-anticipated, most-covered event.

I believe there are far better ways to demonstrate these bugs. FIRST has held Beta Day in Manchester. If the FTA and some volunteering teams were on board, I'd be okay with someone demonstrating a novel flaw the night before SCRIW and forwarding the information to FIRST. And if you feel the urge to break something at Championship, why not practice matches with your own team?

The phrasing of Jon's email makes me believe the individual involved is (well, was) a mentor on a team. As the guy in denim says, we get the best of what we celebrate. There is no room for celebrating interference with any competitive FRC match, whether it's Einstein or Q12 at some Box-On-Wheels Extravaganza of a regional. There is plenty of room to celebrate mentors that discover field issues and disclose them responsibly.

[Ether]

http://www.chiefdelphi.com/forums/sh...&postcount=115

[Ether]

Originally Posted by Billfred

There is no room for celebrating interference with any competitive FRC match, whether it's Einstein or Q12 at some Box-On-Wheels Extravaganza of a regional.

Just to be clear, you're not implying that anyone who has posted to this thread is suggesting that "celebration" is the appropriate response to what happened, correct?

[Ether]

Originally Posted by Chris is me

And does it even matter what his / her intent was?

Yes. Intent matters.

[LeelandS]

Originally Posted by akoscielski3

I actually would prefer this team remain anonymous. Reason is that the Team itself is not responsible (or so it seems). And what ever the mentor/student/parent did wont ruin the whole teams reputation. If the team is revealed the team will become ashamed for what another individual did and could possibly shut down. I would hate to see this happen. Its not the Team's fault and I would hate to see them suffer from it.

Just my $0.02 CAN

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

[Alan Anderson]

Originally Posted by Jedward45

Could someone summarize and explain the more detailed aspects of the Robot Testing and Failed Client Authentication Testing? Specifically, what intentional interference actually happened, how did it cause problems, and what are they planning to do to fix the issue?!

There's a bug in the version of firmware used on the field access point in Week 4 and later. It manifests itself when a D-Link with Rev A hardware is connected to the access point. If a "rogue" client attempts to connect to the wireless network and provides an incorrect WPA security key, the wireless link between the robot and the access point is disrupted. A subsequent failed connection attempt by the rogue can restore the robot link.

The "intentional interference" was someone using a smart phone to try to connect to the network established for one of the teams playing a match and failing. That should have resulted in exactly zero effect on the networks, but the firmware bug caused that team's robot to lose its link to the field network.

As an immediate fix, they're reverting to a previous version of firmware for the access point. Testing showed that version not to be vulnerable to the Failed Client Authentication problem. The ultimate fix will involve fixing the bug, including a test for the problem when doing validation acceptance for new revisions of firmware, and putting specific features in place to detect and log such attempted connections.

[dodar]

Originally Posted by LeelandS

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.
I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

If you did do that, then we wouldnt be having this conversation, which would in turn make you not go and invent a time machine. So seeing as that didnt happen, either you fail in doing so or you do invent a time machine but you get taken out by the duplicate paradox...of course if that happens, which I hope it doesnt, you in essence die the day you go back in time to stop yourself from screwing up at last year's championships.

[ttldomination]

Originally Posted by Chris is me

And does it even matter what his / her intent was? Are the affected teams supposed to feel better about being cheated out of a fair chance at victory because "oh, he / she had good intentions"?

In the grand scheme of things, it doesn't matter. The person was punished, and it appears that it's an acceptable method.

However, intention is something that turns a situation on its head. Intention is what's driving every single post about how appalled people are that this happened. We are assuming intent, and therefore we are writing posts filled with hate.

I will iterate that I cannot begin to imagine how the effected teams are feeling, and the goal of my post was not to curb their anger/disappointment, to that, I have no right. My goal was to simply promote an ounce of open minded-ness.

- Sunny G.

[apalrd]

Since we're (mostly) engineers on CD, let's think of this as an engineering problem. There is a supplier-customer relationship where FIRST is the supplier (of the control system) and the teams are the customers (of the control system).

The customer must have requirements for the supplier. The basic requirements for an FRC control system by a team are:

-Must be easy to setup for a very inexperienced team. Even if the manuals are clear, complex systems are inherently easy to misconfigure even if the directions specify how to do it.

-Must have a boot time of no more than (x) seconds for the entire system. I would be willing to exempt the DS only because I know how slow Windows is. This time limit is to the advantage of teams who want to run quickly, and the field crew which can recover from errors more quickly.

-Must have a certain amount of minimum functionality that can be achieved with minimal computer skills by a team - I would include a requirement for a certain amount of default code, at least mapping JS axis/buttons to motors and solenoids, since I know how many teams previously relied on default code. For some reason, the current control system lacks this and I don't really like it.

-Must be protected from any interference - I believe this requirement previously existed but was not fully met. I will discuss this later.

-Must have certain safety checks implemented - Specifically loss of communication with driver station, crashing of team code, and network error. Currently the first two are not implemented well, the third is implemented with a packet CRC which is good.

Good requirements? I think this covers the basics.

Now how do we verify that we've set all of the requirements correctly? How do we set the exact parameters of the requirements (boot time, amount of sophistication of hacking attempts, default code, watchdog timers, etc.)? Area they created by the supplier arbitrarily, or does the supplier actually know what the proper values are from their experience in designing control systems? Does the supplier actually work with control systems like these, or are they guessing based on experience in another field (say, industrial controls vs automotive controls - Timing and power requirements are *very* different between these two).

There are a few key flaws in this system which are highlighted above. Who is making the requirements that define boot time, maximum time without communication before the robot disables itself, or the minimum sophistication of a hacker attempting to compromise the security system? Do the requirements even exist, or is it more of a best-attempt system for some of these variables?

Best-attempt parameters always ends up as a variable which can be compromised to meat any other goal, with no real loss to the supplier.

The next step in engineering is actually designing the system. So, let's skip that step and end up with what we have now.

Once the system is engineered, you have to verify that you have met all of your requirements. How do you test this? While nobody thought to test the case of 802.11 access requests stated in the document, and this is OK given the obscurity of the bug and it's existence in hardware from another supplier, I can guarantee that somebody thought about DOS attacks on 802.11 (if they didn't, they would have not come close to meeting the requirement set above). What is the solution to them? The AirTight device was clearly chosen to detect DOS attacks, without adequate testing to verify that it actually met the requirements.

I am not pointing blame to anyone or any company specific, as I know there are many people and companies involved in the design of the system, but it has issues on a whole that are nobody's fault.

This document clearly shows me why 802.11 is not used in critical applications in this way (open SSID broadcast especially). No offense to FIRST, but the choice for 802.11 is probably the largest design failure of the entire control system (including my rants on compile/download/boot times and stuff). This testing, specifically the individual who is anonymous, shows just how vulnerable the system is, and how little it is protected from any intention or unintentional interference from a device that everyone carries in their pocket.

As a final thought, if the issues are blamed on the teams in such large quantities, then something must be too complicated, ambiguous, or otherwise error-prone in the system (basically the system is too complicated if that many people can't get it setup correctly).

Someone earlier in the thread questioned the failure at GTR as possibly being a related event. There were more failures than just GTR (although it was the most publicized) that showed up in exactly the same way as what happened to Simbotics. Teams' radios were blamed by FTA's, and everybody lived unhappily because the radios did suck (they thought) and there was nothing they could do to change it. The FMS was "above blame" - Because the DS could not ping the teams radio but the other 5 DS's could ping the other 5 radios on the field, it must have been the fault of the team's radio. (on a related note, the FMS gathers very little logging data for itself, most of it is collected by the DS and forwarded to the FMS). The general idea is that, since the cause isn't definitely the field, it's probably the teams (especially since so many teams have so many issues with the radio power wiring) and then it becomes always the teams fault, and the FMS is never blamed for failure of communication.

Anyone want to suggest another air interface? I've been thinking about a few...

[Chris Fultz]

This (similar) post is in the thread with the FIRST letter and link, but i was requested to duplicate it here -


There will likely be several threads and posts about the report and the contents. Please remember this is a public forum, use caution and care in what you say and what you claim.

Also, please read the report before commenting on it. There is a lot of information, and many questions and comments may be answered in the report itself.

Regarding the report, this is a detailed summary of the fact-finding, the process, the testing and the results and conclusions found by a large, diverse team. You may or may not agree with all of the conclusions drawn, but there is a great deal we can all learn from the detail of the report.

We also owe a lot to the 12 teams involved and their level of participation at the FIRST weekend and data collection process.

Also, note the request from FIRST for input. If you have constructive input, please use the email address to provide that to FIRST.


* Addition -

Read the report and the detail for what it actually says, not for what you think it says. And, to keep the thinking clean, if someone make a conclusion or statement not supported by the report, then call them on it and clear it up. Some are doing this already and it keeps the conversations and conclusions accurate.

[cgmv123]

Originally Posted by apalrd

Since we're (mostly) engineers on CD, let's think of this as an engineering problem. There is a supplier-customer relationship where FIRST is the supplier (of the control system) and the teams are the customers (of the control system).

The customer must have requirements for the supplier. The basic requirements for an FRC control system by a team are:

-Must be easy to setup for a very inexperienced team. Even if the manuals are clear, complex systems are inherently easy to misconfigure even if the directions specify how to do it.

-Must have a boot time of no more than (x) seconds for the entire system. I would be willing to exempt the DS only because I know how slow Windows is. This time limit is to the advantage of teams who want to run quickly, and the field crew which can recover from errors more quickly.

-Must have a certain amount of minimum functionality that can be achieved with minimal computer skills by a team - I would include a requirement for a certain amount of default code, at least mapping JS axis/buttons to motors and solenoids, since I know how many teams previously relied on default code. For some reason, the current control system lacks this and I don't really like it.

-Must be protected from unauthorized interference - I believe this requirement previously existed but was not fully met. I will discuss this later.

-Must have certain safety checks implemented - Specifically loss of communication with driver station, crashing of team code, and network error. Currently the first two are not implemented well, the third is implemented with a packet CRC which is good.

Good requirements? I think this covers the basics.

I think you're forgetting advanced functions, speed and customization for advanced teams. It can't be so simple that advanced teams can't take their robots to the next level.

[Gigakaiser]

Originally Posted by LeelandS

I agree. Not just because the team's reputation will be almost unquestionably destroyed. Though that would be a terrible shame. My biggest concern is that the person would be CRUCIFIED. Metaphorically. Hopefully not literally. I like to believe FIRSTers aren't that kind of people But anyways. The person is already banned from FIRST. If that were me, I don't know what I'd do with myself. I'd probably devote my life to inventing a time machine so I could go back in time and punch myself in the face before I messed up Einstein.

I'm sure (or at least, I hope) this person is truly remorseful for their actions. If they have seen a fraction of these responses, I'm sure they'd know that their actions deeply upset a large number of people. The last thing this person would need is to be forever known as "The person who ruined Einstein." If their identity were to become public, let's face it: No FIRSTer in the world could look at them the same way. They would be faced with eyes of raw disdain and disappointment. All respect from the FIRST community would be lost, or at least severely damaged. I, personally, don't think anyone deserves that.

I disagree - the team (or whoever on the team is aware of the individual) should release the information and acknowledge that it was an individual, not a team effort. Imagine the reputation of the team if the information came out sometime in the future by another source - it would not look good. If the team came out it would be seen as a gracious step forward. I highly doubt the entire team was aware of the individual during einstein anyway.

[Alexa Stott]

Originally Posted by Ether

or her. The report does not identify the gender of the individual.

The male pronoun is often used in cases of ambiguity. That you took the time to take issue with this incredibly trivial part of my post indicates that maybe we all need to step away from our keyboards for a bit.

[dodar]

Originally Posted by Gigakaiser

I disagree - the team (or whoever on the team is aware of the individual) should release the information and acknowledge that it was an individual, not a team effort. Imagine the reputation of the team if the information came out sometime in the future by another source - it would not look good. If the team came out it would be seen as a gracious step forward. I highly doubt the entire team was aware of the individual during einstein anyway.

Wouldnt matter. From then on that person would not be known for their specific name but as a member/former member of team _______.

[Gregor]

Originally Posted by Joe Johnson

I don't want to start a rumor but does anyone else think that this must be related to the nonsense that went on at the Greater Toronto East Regional? If so, the Canadian FIRST community really has to work to lance this boil.

If you honestly believe that you can draw parallels between not attempting the coop bridge with very very good teams and deliberately sabotaging the premier FRC matches, I don't even know what to say... You can tell that there is some animosity towards these outstanding teams if you went to a Canadian regional, but if you think that someone wants to ruin their Einstein appearance for some twisted payback, I think you're sadly mistaken.

Please don't make generic assumptions when you have absolutely no proof, only speculation on whom it may have been. You know what they say about assumptions...
---

I believe that FIRST staff, volunteers, and Einstein teams handled the situation exceptionally well given the circumstances. Lets not forget that 12 incredible teams went onto the Einstein field, and 12 incredible teams left it.

For this one person who deliberately tried to sabotage the event, there are literally hundreds of thousands of people who would condone in. Lets remember that one rogue person shouldn't be regarded as a significant change in FIRST culture.