[FRC Blog] Einstein Report Released

[Al Skierkiewicz]

Justin,
It is my understanding that your code has a method for checking all sensors as part of the initiation sequence. When the gyro reported bad, the code stopped. We tested this in NH and received the same result when the gyro wire was pulled. During Einstein, we replaced your Crio, DSC, DSC cable and everything else I could think of without effect. We checked for power issues, bent cables, as much as we could in the amount of time we had. I have to tell everyone that 1114 and others were offering hardware and assistance during this time as you would expect. (remember Thunder Chickens from a few years ago?) The FTA and CSA came over to lend support as well.

[Al Skierkiewicz]

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

[Sam124]

Originally Posted by Al Skierkiewicz

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

I couldn't agree more with every point that you make, Al.

[KelliV]

Hi guys, I've waited awhile to post so that I could clear my head and here goes.

Al said pretty much everything I would want to say about the intentional interference so I won't even address that.

I do, however, have a suggestion for those of you who are significantly smarter when it comes to technical mumbo jumbo that I am. Several of my friends have looked at the report and come away with the same reaction as I have, that being... huh???

To help others understand what really went wrong can someone sum up, in easier terms, what happened technically with the robots. Imagine what you would say to someone who isn't in FIRST when explaining what's wrong. I think this is part of the problem as to why many are seeing, as Dr. Joe put it, the tree rather than the forest.

Everyone out there understands what happens when someone interferes, but not many people other than FIRSTers really understand CRio problems or CPU usage. So on that long drive to IRI if someone could make sense of a lot of this it would really help a lot of people out.

[simpsonboy77]

KelliV,

I'll try to simplify it as much as possible.

The first tests (1-6) were to test FMS. There could be many issues with the hardware used to run FMS, so this was to test the software FMS uses, hardware, and peripherals. I don't know exactly what they were looking for, but I'm guessing just stability issues. A faulty router could cause drop outs etc. Basically you are looking for something, you don't know exactly what it is, but when you find it you will know. This is the approach for intermittent issues, poke the system until something gives.

The client authentication issue works due to issues in the D-Link DAP-1522 Revision A hardware. This is the wireless bridge that you stick on your robot. This can only manifest itself when paired with the newest firmware of the access point used during competition. No team would have control over this as it is FMS controlled. It is a Cisco 1252. The firmware was changed in week 4 because of an issue where the 1252 would reboot while setting up a new match. I am going to guess that this was done to decrease setup time between matches. However Cisco did not verify the setup in entirety. Had they done this, the issue would have been caught, and likely fixed in firmware.

As for which device connects, it honestly does not matter. What matters is how often the device connects to the network. It is relatively easy to write a program that will run on a laptop to imitate this. What is worse is such a program can be written and run without user intervention, so the person next to you in the stands would not know if you are attacking the field. I will skip what happened at Einstein as that is not as technical, and I am somewhat pressed for time making this post.

De-auth attacks are very effective at causing denial of service. Basically an attacker can spoof the access point, and tell the robot's bridge to reauthenticate and associate. This takes time, and it is trivial to send this packet upwards of 400 times a second at one robot. I may have misinterpreted this, but an attacker can send 400 packets a second at a single robot for 89 seconds and not be detected. Additionally an attacker can trickle deauth packets for the entire match and not be detected. A trickle of 3-7 packets per second will cause some network congestion for that one robot. This is not so much a vulnerability in any hardware or FMS, but more so a vulnerability in 802.11. There are other vulnerabilities such as TKIP session hijacking in TKIP (we use AES), and hole 196 in WPA2 AES. It becomes very difficult to fix these bugs as they are beyond the scope of FMS and FIRST.

The robot specific issues are to show that while it may have been an attack the entire time, these other issues likely contributed to failures.

A high CPU usage could hinder the cRio's ability to send packets to communicate with the driver station. If the cRio is trying to process images, that is time it is not using to communicate with the driver station. If enough data gets buffered and not handled, this will cause drop outs. Think of it as someone reading off several phone numbers and you are writing them down. If the person reads them too fast, you cannot remember all of them. The ones you forget are analogous to dropped packets.

Further down in the report they mention QoS, or quality of service. This will let FMS limit how much bandwidth a robot can use on its own. There is ideally only 300 mbps of bandwidth available amongst 6 robots. Two uncompressed video feeds can eat up much of this. This will prevent robots from starving other robots of bandwidth. It can also be configured to prioritize driver station packets over custom and video packets. This will likely be done off of ports, however I do not know for certain.

As for additionally monitoring, computers can be setup with programs such as wireshark to listen to ALL communication. This will be useful to spot attacks that aren't listed and provides comprehensive logging. Wireless cards can sometimes be put into a mode call promiscuous mode which allows them to listen to data even not directed at them.


I hope that helps, I tried to clarify as much as possible. The report was extremely thorough. I read it first and thought it was missing some important data points, however on a second and third read through, I realized I just missed them >.<

-Garrett

[Greg McKaskle]

Garrett's email explained things pretty well I thought. If there are more specific questions, this is a pretty good place to ask.

Regarding CPU usage, the code was inspected to ensure the reason was well understood. 100% usage itself will not cause communication issues, but it may be a symptom of something else. The DS logs of each robot for each Einstein match was also inspected for CPU, battery, communication quality, timing of dropouts, and other odd patterns in how control packets were processed.

Greg McKaskle

[BigJ]

Originally Posted by simpsonboy77

snip

Good explanations of concepts, but a quick glossary:

Spoof: When a client or user "spoofs" another, it provides the communication destination with information that leads the destination to believe that the client or user is one that they are not.

Packet: Information gets sent over the network in packets. Slow or missing (also called dropped) packets make the driver<->robot appear slow (or laggy) or even disconnect.

[EricVanWyk]

The super simplified version:

Robot: Hey FMS, want to chat? Here is my key.

FMS: Looks good! Lets chat!

"Individual": Hey FMS, want to chat? Here is my key.

FMS: Hey, that key is wrong. I don't want to talk to you.

Robot: Aww, the FMS doesn't want to chat with me anymore. Was it something I said?

[Anupam Goli]

Originally Posted by EricVanWyk

The super simplified version:

Robot: Hey FMS, want to chat? Here is my key.

FMS: Looks good! Lets chat!

"Individual": Hey FMS, want to chat? Here is my key.

FMS: Hey, that key is wrong. I don't want to talk to you.

Robot: Aww, the FMS doesn't want to chat with me anymore. Was it something I said?

In the seriousness of the discussions going on, this one seriously made me laugh, and is a great analogy.

[JaneYoung]

These last few posts that took the time to answer Kelli's request - are proving very helpful. Thank you so much.

Jane

[EricVanWyk]

Super Simple Deauth:

"Individual": I am the FMS. I don't want to talk to you.

Robot: FMS, is this true? I still have stuff for you.

Actual FMS: No, what gave you that idea? Are you feeling ok?

Repeat.

Field Monitor: FTA! We need a counselor!


Result: Communication slows down, and can sometimes drop out entirely. This is a possible attack that was largely ruled out. If someone were to launch this attack, the Airtight system would count the number of "go away" messages and display an ugly warning when it hit a certain threshold. We found that that threshold was too loose, so we're tightening the chain. We can't set the threshold to 1, or we'd get false positives several times a match - it is a valid message to send.



Super Simple Priority Inversion:

Robot: Gyro, please reset yourself and verify proper operation.

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...

Field: Robot, I keep telling you to reboot, but there there is nowhere for me to put my message to you.

Result: The robot code locks up. The cRIO's safety mechanisms kick in and prevent the bot from moving. This is actually a good thing. The bad part was that it wasn't able to get the command to reboot and try again.


Super Simple Network Tables Flood:

Dash Board: Hey Robot, here is some new data.

Dash Board: Robot, did you get that data yet? Hello?

Robot: Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks!...

Other Robots: Will you please shut up?

Result: A few seconds of extra lag. All of those unnecessary acknowledgements eat up radio time, and can cause brief control losses. The plan is to put fairness guarantees in place that prevent this type of error from affecting the other bots.

[Racer26]

@EricVanWyk: AWESOME analogies. Almost totally accurate too.

[Akash Rastogi]

Originally Posted by EricVanWyk

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...

This might be the funniest thing I've ever read on CD. Thank you.

[Greg McKaskle]

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

[BigJ]

Originally Posted by Greg McKaskle

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

Who watches the watchdogs?