Re: Team 548 Einstein Statement
 

No one I know that has so far commented has gotten so much as an auto response (a courtesy).

In 17 years my experience calling the FIRST switch board is dismal.

Asking questions in the actual Q&A forum has often been criticized above and beyond this point (to the point I know people who intentionally avoid it).

My experience obviously differs from your own.

You might consider it not a threat to make such a disclosure with lots of time to resolve it, but under the current circumstances I see nothing, at all, that prevents FIRST from viewing your eventual disclosure as an open challenge to their authority.

Right on topic the last person that pointed out something was asked to leave.
One could argue that it would have been subsequently followed up.

However, no where in any discussion that I have seen (or the reports) did it indicate what the process for that follow up was or was ever outlined to the reporting party.

So I bring this back full circle. There are disclosures of issues I am aware of. What is the process by which these courtesies are reciprocated? I posed that same question weeks ago as well.

Re: Team 548 Einstein Statement
 

One perspective that I think has not been brought up, that I think deserves attention is the competition rules. [T14] states:

"If a team needs clarification on a ruling or score, a pre-college student from that team should address the Head Referee after a field reset has been signaled. An team signals their desire to speak with the Head Referee by standing in the red or blue Question Box which will be placed on the floor at each end of the scoring table. Depending on timing, the Head Referee may postpone any requested discussion until the end of the subsequent Match."

While that does not mention the FTA, it is the closest thing I could find to how an official interaction is made concerning the results of a match. I'm not saying this would have affected how staff reacted but I'd like to point out that, from my interpretation of that rule, the proper way for the mentor to bring this up at the field is not at all. If (s)he wanted, (s)he could have revealed this vulnerability to a team member, the team member would have stood in the question box and voiced these concerns with to the Head Referee, who would (hopefully) confer with the technical staff present, and things could have played out differently. I'm not saying they necessarily would have, but we do have rules about who engages field staff, it clearly indicates that only pre-college students may do so, and I know, when I'm volunteering on the field, I would rather talk to a student than a mentor.

Re: Team 548 Einstein Statement
 

I've been watching this thread with much interest lately, and a few interesting points that (I believe) have not been addressed are still fresh in my mind.

First, aren't we forgetting the second person who brought down communications? The story that is corroborated both by the 548 mentor and the official report implies that there was a second attacker, who interestingly attacked the wifi network only after the 548 mentor did his three second demo attack. Most people appear to be assuming that the 548 mentor did all of the wifi atacks, which just doesn't appear to add up. Why did the second attacker act? Did they believe something similar to the first attacker, that they were being attacked? Or did they simply have a malicious intent?

Second, was there institutional knowledge of this security hole? It appears that at least two (and probably more, if this thread is any indicator) FRC members knew of this specific hole. Did no one on the official FRC team know of this? This seems unlikely to me, but depending on the extent of the knowledge of this hole, it certainly could be true. If so, why didn't they attempt to patch it? If not, does this point to an institutional problem in a lack of focus on security? In either case, more needs to be done to recognize and address future security holes.

Third, why did we never learn about this hole at Einstein, where it's relativity unlikely that two separate people coincidentally used this technique to bring down a match. Were there smaller incidents at regionals and division championships that simply did not get noticed until Einstein? Were people with knowledge of this quite until then, or simply unnoticed? And why did a thread never appear on CD with information about this? Surely, unless there was malicious intent, any loyal FIRSTer would rather report this than use it in a match. Were malicious (or simply very quite) people the only ones who ever knew or suspected a exploit of this type?

Hopefully, my questions were constructive and not offensive. I'm just a little surprised that I've never seen them asked or answered yet.

Re: Team 548 Einstein Statement
 

You are forgetting one thing: T14 ONLY addresses Ref interaction! So your interpretation is that the head ref is the only person on the field that questions can be asked of. Have you or any member of your drive team asked a field resetter anything? How about discussing why your robot isn't connecting with the FTA or FTAA? I'm so sorry, but by your interpretation, you just did something illegal. Move along, you can't discuss that with that person.

Now, would it have been helpful to send a message by that route? Maybe--but that involves a) finding a student who isn't trying to fix something and b) having said student wait until they could get the head ref's attention. Then the head ref has to decide that it's important enough to call the FTA or FTAA away from whatever he's doing (probably trying to fix the problem with 118, in the case of 548's matches), oh and did I mention that by now it's second-or third-hand informationsuspicion (which, if you're paying attention, you may have figured out that that's roughly equivalent to a rumor). In other words, chances are fairly high that going that route you'll either be ignored, or if you do get through, the FTA will want to talk to the originator (in this case, the mentor), and we're right back where we started.


@DampRobot: I didn't pick up the implication of a second person involved in the official report. I got that only from 548's account. Also, a 3 second attack like that one would result in needing to reconnect the wifi, which can take a little bit of time, regardless of if there's another attacker or not. I think a lot of the questions you have are going to be very difficult to answer without putting people under suspicion of cheating or of total ignorance, either of which I'm reluctant to do.

Re: Team 548 Einstein Statement
 

While you bring up good points, are you underestimating how difficult this was to purposefully discover and/or how lucky you'd have to be to find it? I honestly don't know, but as I understand it the Cisco firmware with the hole only implemented in Week 4, and even then only manifest in one of the D-Link revisions. While FIRST tested the new firmware thoroughly for the issue it was meant to address, it's not so surprising they didn't test for FCA (page 7). Conceding (as the wireless experts did) that it's not an obvious issue to test for, I'd be somewhat surprised if FIRST officials managed to trip on it in the intervening weeks. Granted, this definitely isn't my area of expertise.

I missed any implication of a second person in the Report. Where are you referring?

I certainly don't take T14 to be the only allowable interaction (having talked to enough FTAs in my day), but it is the only guaranteed interaction. While I've never done it on Einstein, I head refs--even busy ones--seem listen to polite students in the box. I think you'd be hard-pressed to find a ref that wouldn't listen twice to "I know what's wrong; please let me show you how anyone in the stadium can shut down any robot on this field". As I understand it, the demonstration is rather quick (pull up the network list and show you can send a client authorization). If so, the student could show this directly to the ref for added clout.

I know what's done is done, but hopefully an earnest examination will help anyone thinking of doing something like this in the future. No matter how helpless you feel thinking someone else is targeting your team, there are always other ways. In fact, you can't count on anyone even listening to you, much less getting a replay, if you try to interfere yourself. (Not that this is the key reason against interference.)

Re: Team 548 Einstein Statement
 

There was no evidence of a second attack. The original attacker suspected that other failures (for known and documented reasons) were being caused by the attack method that had been discovered. As to the three second attack, please read the report again! Once a device had attempted to communicate with a robot, the disruption could last the entire match. The attacker could easily move on to another robot(s) after the first disruption.
Also note, the robot remained connected to the field and in those cases where the team was using video from the robot, all status and video continued to be displayed at the driver's station. The robot was connected, just the command link from driver's station to robot was interrupted.
There was no knowledge of this weakness prior to the mentor coming forward and explaining what had actually taken place after the Champs. The mentor was observed on Einstein doing something suspicious with a phone. Anyone repeatedly punching a phone within feet of Einstein while a match is going on is suspect because they are not observing the match at hand. However, the problems did not take on the typical signs of a DOS attack. Had anyone been knowledgeable of the hole (or if the problem had been communicated to the engineering staff), a simple revert to previous firmware, a change in wireless access points on the robot or a combination of the above would have simply fixed the issue. Those changes could easily be made during other closing ceremonies.

If others knew or suspected an issue at other events, they did not come forward with that info. The Einstein Investigation had a clear set of goals and that was to determine what caused so many failures on the Einstein Field. We were not tasked with investigation outside of Einstein and the twelve robots involved in that part of the competition.

To be absolutely clear, there are many people on or near the field during events. Some of these are non-technical volunteers and some have been tech volunteers in the past and some are volunteers who are also on teams competing on the field. Approaching one of those volunteers and expecting the same response as a field expert to a technical issue like this is a bad use of time. At every event there is a crew of volunteers whose directive is to make every robot play, that is the Robot Inspectors. During Champs finals, (all divisions and Einstein) there are inspectors assigned to the field to assist teams with problems and work with the head referee and FTA. There were two experienced division LRIs on Einstein, one on each side of the field during the matches and in the pit area assisting teams between matches. If you have a problem and cannot get resolution, please check in with an inspector or LRI. We want everyone to play, as often as they wish, within the rules of the competition.

You wouldn't necessarily have to know the cause of the issue to happen upon the exploit. With the growing number of applications that can control any number of robots with a smartphone, it's really not surprising that between week 4 and Einstein someone whipped out a phone and thought, "What if I connect in during a match?"

It's the "1000 monkeys with 1000 typewriters" postulate at work, and I think it would be wise of FIRST to challenge all teams to try and find these exploits and notify FIRST as they appear. Crowd-source the troubleshooting of these systems, and allow teams to have active feedback throughout the season. It would solve a lot of problems. And I agree with the idea that FIRST should have some kind of pre-written response to let teams know that emails are at least going through.

Re: Team 548 Einstein Statement
 

That's a great idea in theory. In practice, however, FIRST would be completely overwhelmed with nonsense results from uncontrolled situations that bear little or no relevance to a competition field setup.

Simply put: the problem with the "1,000 monkeys with 1,000 typewriters" postulate in reality is filtering out the 99%+ gibberish content they've created.

Re: Team 548 Einstein Statement
 

To "happen upon the exploit" requires specific hardware. If someone had tried to connect without using one of the exceedingly few handheld devices capable of 5 GHz WiFi, nothing would have happened. That's a good enough reason for me to accept the idea that nobody but the admitted culprit knew about the problem.

Re: Team 548 Einstein Statement
 

The simple way to find the non-gibberish is request a proof of concept either in video or in front of field personnel.

This would be easier to accomplish with more open documentation about the field (so it can be more readily replicated) and more access to fields (itself not a trivial request).

Of course all of that is useless without clear lines of communications and process.

Also there are probably more devices than one might realize at any one event that can use 5GHz because they are not line of sight to the field. Consider all the driver's station laptops in the pits. I'll assume that no one on the field with a 5GHz laptop has time to be doing anything but what is expected of them.

With Windows Vista and above it would be very simple to craft a background script running as system that would exploit the failed connect attempt hole totally hidden from all but the most experienced eyes even on a driver's station on the field (in effect malware for the field). This wouldn't seem out of place at all because of the driver station software reliance on Windows. Also if someone had a COTS computing device on the robot a similar tactic with wider OS selection would be possible. I am comfortable making this statement because this particular vulnerability is much easier to remedy than others I am aware of.

Re: Team 548 Einstein Statement
 

Al Skierkiewicz, thank you for pointing out that what might seem obvious to me might be completely contrary to others' points of view. To address your comments using my interpretation of the report:

First, the official FRC report describes a Galxey Nexus running Android 4.0.4 was probably used for at least one attack ("Failed Client Authentication on Einstein") that we recently learned was committed by the 548 mentor. Another section of the report ("Alternative Source Testing") describes in detail the attempts to bring down communications with the failed client authentication attack, and that downtimes in communications could be as low as three seconds with that device and by using a specific strategy. Especially if the mentor had tried this before (which I'm certainly not trying to imply!), he certainly could have only brought down communications for only three seconds.

The second attacker was, to me, implied by the fact that the mentor left the field before Final 1 and 2 and that continued attacks occurred. Also, witnesses saw an individual selecting teams to take down from a cell phone, who may or may not have been the same mentor. Although they believe they are one and the same, the mentor repeatedly denies doing this attack more than once (and if he had, why wouldn't he have used the strategy that would have resulted in only 3-second downtimes? Malicious intent?). He certainly may have been lying, but the fact of the continued attacks considerably longer than three seconds and their continuance even after this person left the field remains.

I think the question of whether there was knowledge in FIRST about this type of hole is a fair question. It states in the Eisenstein report that they only discovered this error accidentally after championships. Shouldn't the actions of this individual, as well as their attempt to contact field personal, given them at least a hint that something was up? Did someone know about this, and was not heard? I certainly don't know, and I don't really expect that anyone on CD can answer all of my questions conclusively.

As always, no offense meant. Hopefully my comments are seen as constructive.

Re: Team 548 Einstein Statement
 

The number of driver station laptops in the pits capable of 5 GHz WiFi was vanishingly small. As a robot inspector, checking for wireless networking of teams' laptops was part of my job. I saw exactly zero with 5 GHz radios in three regional competitions and a championship division.

Re: Team 548 Einstein Statement
 

Fair enough but it can be added in a second with a USB port or card if they choose. Also what about the other laptops often in the pits:

Apple laptops, most all of them since 2006, have dual band.

Including the MacBook, the MacBook Pro, and the MacBook Air.

I know I saw a few of those in my trips into the pits at various events even if they weren't driver's stations.

Re: Team 548 Einstein Statement
 

Damp,
The three seconds referred to in the report is the response to a specific set of steps taken and observed by the First engineering team testing the Samsung Galaxy Nexus phone at HQ. It is not suggested that this is what action was taking place on Einstein, merely an additional failure using that phone during testing. The alternative testing was performed after it was noted that a 5GHz enabled wireless device had caused some issues on Einstein. It was noted by First engineering that devices have this tendency to 'phone home' once they see a wireless network that they recognize. That is the "repeat interval" listed in that part of the report.
In addition from the report..."Each of these authentication attempts has the potential to cause working communication to drop and a dropped connection to be reestablished between the driver station and the robot. Repeated attempts to connect to multiple SSID’s can result in robots that are drivable and robots that are not over the course of the match."

Re: Team 548 Einstein Statement
 

I agree with you--in "1000 people" [likely more] that were around fields on/after Week 4, it seems somewhat plausible to me that someone else who happened to have 5GHz WiFi happened to try to connect to a robot who happened to have Revision A, and happened to try entering a password and cause FCA, and happened to be one of the people that would keep it to themselves. Not likely, but plausible.

What I find significantly less plausible is that FIRST officials happened to do so. Not only is the sample size many, many times smaller, but they are naturally quite busy during matches and additionally have every reason to trust in FIRST's testing. (I acknowledge the potential for complacency.) I cannot picture an FTA or FTAA (etc), much less Dean or Woodie, whipping out their phone in the middle of a match. They have every reason to be among the most busy people in the stadium and no reason to distrust their own selections. This is my argument against DampRobot's question of institutional knowledge.