Re: [FRC Blog] Einstein Report Released
 

Justin,
It is my understanding that your code has a method for checking all sensors as part of the initiation sequence. When the gyro reported bad, the code stopped. We tested this in NH and received the same result when the gyro wire was pulled. During Einstein, we replaced your Crio, DSC, DSC cable and everything else I could think of without effect. We checked for power issues, bent cables, as much as we could in the amount of time we had. I have to tell everyone that 1114 and others were offering hardware and assistance during this time as you would expect. (remember Thunder Chickens from a few years ago?) The FTA and CSA came over to lend support as well.

Re: [FRC Blog] Einstein Report Released
 

Everyone,
I have forgotten to mention that anyone who knows (or thinks they know) of other vulnerabilities is asked to send those reports to 2012frcfeedback@usfirst.org.
This is the same address listed in the report. FIRST Engineering is reading through those emails so your input will be a big help in further testing.

Re: [FRC Blog] Einstein Report Released
 

I couldn't agree more with every point that you make, Al.

Re: [FRC Blog] Einstein Report Released
 

Hi guys, I've waited awhile to post so that I could clear my head and here goes.

Al said pretty much everything I would want to say about the intentional interference so I won't even address that.

I do, however, have a suggestion for those of you who are significantly smarter when it comes to technical mumbo jumbo that I am. Several of my friends have looked at the report and come away with the same reaction as I have, that being... huh???

To help others understand what really went wrong can someone sum up, in easier terms, what happened technically with the robots. Imagine what you would say to someone who isn't in FIRST when explaining what's wrong. I think this is part of the problem as to why many are seeing, as Dr. Joe put it, the tree rather than the forest.

Everyone out there understands what happens when someone interferes, but not many people other than FIRSTers really understand CRio problems or CPU usage. So on that long drive to IRI if someone could make sense of a lot of this it would really help a lot of people out.

Re: [FRC Blog] Einstein Report Released
 

KelliV,

I'll try to simplify it as much as possible.

The first tests (1-6) were to test FMS. There could be many issues with the hardware used to run FMS, so this was to test the software FMS uses, hardware, and peripherals. I don't know exactly what they were looking for, but I'm guessing just stability issues. A faulty router could cause drop outs etc. Basically you are looking for something, you don't know exactly what it is, but when you find it you will know. This is the approach for intermittent issues, poke the system until something gives.

The client authentication issue works due to issues in the D-Link DAP-1522 Revision A hardware. This is the wireless bridge that you stick on your robot. This can only manifest itself when paired with the newest firmware of the access point used during competition. No team would have control over this as it is FMS controlled. It is a Cisco 1252. The firmware was changed in week 4 because of an issue where the 1252 would reboot while setting up a new match. I am going to guess that this was done to decrease setup time between matches. However Cisco did not verify the setup in entirety. Had they done this, the issue would have been caught, and likely fixed in firmware.

As for which device connects, it honestly does not matter. What matters is how often the device connects to the network. It is relatively easy to write a program that will run on a laptop to imitate this. What is worse is such a program can be written and run without user intervention, so the person next to you in the stands would not know if you are attacking the field. I will skip what happened at Einstein as that is not as technical, and I am somewhat pressed for time making this post.

De-auth attacks are very effective at causing denial of service. Basically an attacker can spoof the access point, and tell the robot's bridge to reauthenticate and associate. This takes time, and it is trivial to send this packet upwards of 400 times a second at one robot. I may have misinterpreted this, but an attacker can send 400 packets a second at a single robot for 89 seconds and not be detected. Additionally an attacker can trickle deauth packets for the entire match and not be detected. A trickle of 3-7 packets per second will cause some network congestion for that one robot. This is not so much a vulnerability in any hardware or FMS, but more so a vulnerability in 802.11. There are other vulnerabilities such as TKIP session hijacking in TKIP (we use AES), and hole 196 in WPA2 AES. It becomes very difficult to fix these bugs as they are beyond the scope of FMS and FIRST.

The robot specific issues are to show that while it may have been an attack the entire time, these other issues likely contributed to failures.

A high CPU usage could hinder the cRio's ability to send packets to communicate with the driver station. If the cRio is trying to process images, that is time it is not using to communicate with the driver station. If enough data gets buffered and not handled, this will cause drop outs. Think of it as someone reading off several phone numbers and you are writing them down. If the person reads them too fast, you cannot remember all of them. The ones you forget are analogous to dropped packets.

Further down in the report they mention QoS, or quality of service. This will let FMS limit how much bandwidth a robot can use on its own. There is ideally only 300 mbps of bandwidth available amongst 6 robots. Two uncompressed video feeds can eat up much of this. This will prevent robots from starving other robots of bandwidth. It can also be configured to prioritize driver station packets over custom and video packets. This will likely be done off of ports, however I do not know for certain.

As for additionally monitoring, computers can be setup with programs such as wireshark to listen to ALL communication. This will be useful to spot attacks that aren't listed and provides comprehensive logging. Wireless cards can sometimes be put into a mode call promiscuous mode which allows them to listen to data even not directed at them.


I hope that helps, I tried to clarify as much as possible. The report was extremely thorough. I read it first and thought it was missing some important data points, however on a second and third read through, I realized I just missed them >.<

-Garrett

Re: [FRC Blog] Einstein Report Released
 

Garrett's email explained things pretty well I thought. If there are more specific questions, this is a pretty good place to ask.

Regarding CPU usage, the code was inspected to ensure the reason was well understood. 100% usage itself will not cause communication issues, but it may be a symptom of something else. The DS logs of each robot for each Einstein match was also inspected for CPU, battery, communication quality, timing of dropouts, and other odd patterns in how control packets were processed.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

Good explanations of concepts, but a quick glossary:

Spoof: When a client or user "spoofs" another, it provides the communication destination with information that leads the destination to believe that the client or user is one that they are not.

Packet: Information gets sent over the network in packets. Slow or missing (also called dropped) packets make the driver<->robot appear slow (or laggy) or even disconnect.

Re: [FRC Blog] Einstein Report Released
 

The super simplified version:

Robot: Hey FMS, want to chat? Here is my key.

FMS: Looks good! Lets chat!

"Individual": Hey FMS, want to chat? Here is my key.

FMS: Hey, that key is wrong. I don't want to talk to you.

Robot: Aww, the FMS doesn't want to chat with me anymore. Was it something I said?

Re: [FRC Blog] Einstein Report Released
 

In the seriousness of the discussions going on, this one seriously made me laugh, and is a great analogy.

Re: [FRC Blog] Einstein Report Released
 

These last few posts that took the time to answer Kelli's request - are proving very helpful. Thank you so much.

Jane

Re: [FRC Blog] Einstein Report Released
 

Super Simple Deauth:

"Individual": I am the FMS. I don't want to talk to you.

Robot: FMS, is this true? I still have stuff for you.

Actual FMS: No, what gave you that idea? Are you feeling ok?

Repeat.

Field Monitor: FTA! We need a counselor!


Result: Communication slows down, and can sometimes drop out entirely. This is a possible attack that was largely ruled out. If someone were to launch this attack, the Airtight system would count the number of "go away" messages and display an ugly warning when it hit a certain threshold. We found that that threshold was too loose, so we're tightening the chain. We can't set the threshold to 1, or we'd get false positives several times a match - it is a valid message to send.



Super Simple Priority Inversion:

Robot: Gyro, please reset yourself and verify proper operation.

Gyro: ... WHY IS THE EVERYTHING SPINNING SO FAST? HELP! ...

Robot: Are you working yet? How about now?

Vision Processor: Hey Robot, here is a whole lot of data for you.

Robot: Not now, I'm still waiting for Gyro to tell me he is ok. Put it in my mailbox, I'll get to it as soon as Gyro is ok.

Gyro: ... *puke* ...

Field: Robot, I keep telling you to reboot, but there there is nowhere for me to put my message to you.

Result: The robot code locks up. The cRIO's safety mechanisms kick in and prevent the bot from moving. This is actually a good thing. The bad part was that it wasn't able to get the command to reboot and try again.


Super Simple Network Tables Flood:

Dash Board: Hey Robot, here is some new data.

Dash Board: Robot, did you get that data yet? Hello?

Robot: Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks! Yep! Thanks!...

Other Robots: Will you please shut up?

Result: A few seconds of extra lag. All of those unnecessary acknowledgements eat up radio time, and can cause brief control losses. The plan is to put fairness guarantees in place that prevent this type of error from affecting the other bots.

Re: [FRC Blog] Einstein Report Released
 

@EricVanWyk: AWESOME analogies. Almost totally accurate too.

Re: [FRC Blog] Einstein Report Released
 

This might be the funniest thing I've ever read on CD. Thank you.

Re: [FRC Blog] Einstein Report Released
 

There was discussion of a graphic novel form of the report. Eric, I think you are off to a good start.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

Who watches the watchdogs?

Re: [FRC Blog] Einstein Report Released
 

I'd rather EJvW draw some (xkcd-like) stick figure drawings in the page corner so we can just flip the pages really quick to animate the sequences.

Re: [FRC Blog] Einstein Report Released
 

THAT would be so epic... :D

Re: [FRC Blog] Einstein Report Released
 

I suppose one should ask what they should expect for a response if they do this?

Re: [FRC Blog] Einstein Report Released
 

Do we have any volunteers for illustrator?

The weird part? This is how I actually think about the control system. Engineering is so much easier when you give all the components personalities. Then you just watch the drama unfold and try to fix it.

Re: [FRC Blog] Einstein Report Released
 

Thank you very much for the laugh.

I made this for you:

http://i49.tinypic.com/ac9s5.png

Re: [FRC Blog] Einstein Report Released
 

Cute.

This is perhaps the greatest thing I've ever seen on CD.

Re: [FRC Blog] Einstein Report Released
 

From the letter from Jon Dudas -

Re: [FRC Blog] Einstein Report Released
 

So, would you consider yourself a group counselor?

Re: [FRC Blog] Einstein Report Released
 

XKCD already has a character that could play the "Individual"... The Man with the Black Hat.



Note: This is a cropped frame from a XKCD Comic #929 "Speculation".

Re: [FRC Blog] Einstein Report Released
 

Actually, I hear that EVW was an LCPC (Licensed Clinical Professional Counselor) for about 20 minutes last year...

Re: [FRC Blog] Einstein Report Released
 

Black Hat Guy is perfect for that.

...okay, maybe I have a xkcd addiction

Re: [FRC Blog] Einstein Report Released
 

I love this one....
I think it'd be great to have some simple break-down of the report for people. I read it, though a large chunk of it was way over my head- this whole dialogue thing is a great way of explaining it :)

Re: [FRC Blog] Einstein Report Released
 

Someone spotlight this post!!!

You should totally start a webcomic detailing FIRST events in a humorous, yet understanding way.