[FRC Blog] Einstein Report Released

[Ross3098]

Originally Posted by JB987

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

I cant help but believe that the Curie Curse had something to do with the Curie alliance in the finals....

[Nate Laverdure]

If I were the boss, I wouldn't have released this report on Friday the 13th.

[apalrd]

There were several things I got out of this paper, especially as an engineer working on engine controllers:

-The Smart Dashboard had a bug which was exploited which caused a deadlock. While all software has bugs, it should also be tolerant of failure, meaning the rest of the system should have been designed to operate (possibly in limited quantity).

-The Smart Dashboard was mentioned numerous times relating to increased network load, especially the funny 1-byte packets.

-The VxWorks operating system handling of the packet buffer seems exceedingly poor. Many other forms of communication (e.g. some CAN stacks) dump old packets with the same ID when they are added to the buffer, this seems like the right move (at least on UDP).

-The boot time of the cRio was mentioned to be 24s minimum.
--I am currently working with an engine controller that can reboot the application software fast enough to not stall the (Diesel) engine while it is running.

-The nature of 802.11 makes it a poor choice for this kind of wireless communication.

I will not comment on anything else.

[whattsheorder]

Impressed by FIRST's response, my faith in FIRST has been restored. So ridiculous what the hacker did though. I'm curious as to what his relations are to FIRST/any teams..

[Steven Donow]

And the usage of a Galaxy Nexus will only fuel iPhone fanboyism

But in all seriousness, pretty disappointing that someone would do that.

Some slight skimming through shows that this is an interesting read.

But reading the planned fixes/changes has me excited...new radio...more documentation on DS components AND field components...looks like next year will be interesting to watch things unfold as FIRST implements new changes/monitors everything (hopefully) more closely...

[techhelpbb]

I'm happy to see that the electronic power supply issues were considered and annotated. All the testing I did at off season events showed few and far between issues related to the AP/router power supply with some issues with the AP/router power supply (from the battery to the radio power input) taking a bit longer than might be a good idea to reach full output regulation from cold power up (how sensitive the AP/Router is to this is variable and some units might be effected). The time the AP/router supply takes to reach full regulation voltage may leave room for the routers to come up in strange states. This makes sense, all the teams suggesting they fixed their problems when they powered down the AP/router a second time and then back on again after powering up the robot when they initially got on the field (how often it happens though would be quite hard to determine).

The person attempting to connect to the field network is bad news and as long as critical field functions are connected to a public common network FIRST will run this risk. Lucky for everyone this was done line of sight to the effect. Frankly the attack (lack of common sense/failure to communicate intentions) in question could easily have been done any number of clandestine ways that would not have been noticed (I am not going to list them out of concern that someone will try them). Unfortunately it's really easy for someone to create something that will attempt to connect to the field network while aggressively hunting for Internet connections.

I see that they considered the antennas for the field and a few placements of the field equipment. I would have thought they would have tested that further with the robot side equipment power measurements as both sides transmit and receive but apparently that did not happen. Suggestions on how to best optimize the range of the KOP standard robot mounted AP I should hope will find their way into next season.

Overall, I'm satisfied that they've done all they can with what they have as far as a test is concerned. I'm not sure I'm convinced that this one person was the cause of so many headaches however. Einstein was hardly the first time connectivity issues surfaced that were not readily explainable by power supplies issues or programming.

My thanks to FIRST and all those that have worked so hard on trying to make sure this does not happen again.

[LeelandS]

I'm going to keep my personal feelings on the shelf for a while. I need to give it some time before I address more... unsavory... aspects of the situation.

But, what I am overjoyed to say, is FIRST did an amazing job at covering the situation. When I opened the document, I expected a 4/5-ish page summary report of what FIRST had been doing with the Einstein teams the past few weeks. I was pleasantly surprised to find an extremely long, fully detailed report of EVERY test and analysis run by FIRST. Bravo, FIRST. Bravo. You owe us nothing. Yet you went through everything for us. You guys rock!

I'll leave it with this. Do we really need a pound of flesh?
Just sayin'.

[aspiece]

Truck Town Thunder, FIRST Team 68, would like to officially support FIRST in the results as well as applaud them for the way it was handled. Situations like this are unfortunate and it can be difficult determining the best solution to this type of problem. FIRST Team 68 supports FIRST in their decision.

[DominickC]

I'm happy for the FMS Whitepaper.

[Camren]

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

[techhelpbb]

Originally Posted by Camren

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

As I've pointed out before in this topic. It's possible that the power supply that feeds the AP comes up in such a way that causes the router to boot in a state that is not useful.

It's the sort of thing that would depend on what the status of the charge is in the capacitors in the system when you power up. Also it would depend on having a AP a little more sensitive than the median.

I tested a few robots at off season events with tiny oscilloscopes attached to them on the field. Sometimes during a power up it takes a little longer to get to regulation voltage than at other times (we're talking milliseconds max here not seconds).

So it's hard to say that every time you had to reboot an AP it was locked up by a bad processor reset or the exploit. Unfortunately as others have pointed out there are good indicators of the exploit behavior but it wasn't tracked throughout the season.

Hard to say how much was interloper, how much was component malfunction and how much build related failure.

Originally Posted by Nick Lawrence

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

This has come up over and over.

If one assumes that more than one person was aware of the exploit.

There is no assurance that the other people that know are on the team associated with the individual.

In fact the deauth attack (there are 2 issues in the report) could easily have been exploited by anyone anywhere. All you'd have to do is Google it.

So no it's not logical to assume the team had to have known or was the only possible exploiter.

More importantly the individual didn't need to do anything really all that unusual besides be too aggressive attempting to connect to the network.

That's too easy for anyone, even a spectator, to do in mere curiosity.

Once we assume that more than just this individual might be involved who is to say that we can trust that someone somewhere won't do something unfortunate to the individual as it's really the same problem:

1. Everyone who keeps trying to find out is doing something FIRST may not be comfortable with.
2. When someone does find out we don't know how they'll behave.
3. Once the cat is out of the bag we won't know the extent or duration of the consequences metered out to this person.

By actively seeking out this person we are very much doing what we all indicated is bad behavior on their part.

Worse as a community (just like this person's team) we'd take the hit for anyone that went overboard.

We're setting ourselves up for a vicious cycle.

[Ether]

In the last paragraph of page 9 of the report, it states:

Over the course of these tests, FRC Engineering was able to determine how to identify a failed client authentication through the log data recorded in the field access point. However, the configuration of the field access points used during the 2012 FRC competitions, including the matches on Einstein, is such that log data is not retained when the access point is powered off.

Then on page 22 of the report:

Root Cause: Failed client authentication
Mitigation: Field Hardware Logging

Could someone with direct knowledge please confirm/clarify: Is the "Field Hardware Logging" mentioned on page 22 specifically referring to retention of the field access point logs mentioned on page 9?

Thank you.

[Al Skierkiewicz]

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

[techhelpbb]

Originally Posted by Al Skierkiewicz

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

I grant you this was hardly the only issue. However, those initial reports were enough for me to have both versions of the D-Link AP and compare them. I am persistent and that's a mere $200 to get 2 units to test (nothing I found pointed to the issue). Besides off season events do not get a spare parts kit. So the worst case was I had spare radios to offer in case something went down at the off season events. Besides I was testing my tiny oscilloscopes at off season events and they were on the power into the radio so if something had happened (and it didn't) I could offer a replacement as compensation. When I finished testing my oscilloscopes I merely gave some new 1522 AP away in trade for samples of misbehaving APs.

Obviously I didn't spend all that time and money to build those little oscilloscopes because I thought FIRST merely had AP issues (though I admit that while I knew and still know more ways someone could interfere with the wireless I never thought anyone would be that devious or in this case so easily caught).

Still it leaves it out in the open that at some point, perhaps multiple times, someone tested that exploit before they tried to demonstrate it. Additionally, I do agree with what you seem to conclude that this person didn't intend to rig the rankings. Surely this particular tampering is not the only issue and focusing on it too much distracts from the bigger issues we all must face from it. I'm not sure we'll ever know the full extent of what happened without adequate logs.

[Astrokid248]

Originally Posted by Joe Johnson

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

Is all of the above okay? Cos the way I see it, there are four different systems. In terms of the goal of FRC, we're still doing well, if not better. That's what I call the Chairman's system. Also, the lengths FIRST went to after championships are quite impressive, in what I guess is the organization system. Then, we have the system comprised of the elements of game play, both on the field and in the pits, and all's that proves is how human we are. The Astros sucked once they got to the World Series in 2004, but nobody says that the system is broken (well, at least not for teams sucking at that high a level). Finally, there's the robot connection system, which is the only broken system out of the group. And that's the forest Lil Lav is talking about, with the tree being our unknown attacker. It's not the guy's intentions; it's his actions and how repeatable they might be, how much further a future attacker can go if we don't plan for these things in the future. And it's a tricky line to walk. Do we outright ban cellphones? If you do, all the teams who use social media from the field side or can't afford cameras beyond their phones are being denied a large portion of their team identity. But clearly letting everyone have access is also an issue. So do we fix it on FIRST's side? Make the routers inaccessible? Because that's got a whole bunch of other issues associated with it. So let's talk about those issues, and find a solution to the problem of a guy with a smartphone knocking out robot connections, instead of finding a guy we can yell at for weeks on end.