[FRC Blog] Einstein Report Released

[connor.worley]

I am definitely happy to see first taking steps to resolve the issues they discovered, especially with better documentation for advanced coding, the investigation of a new radio, and a fix for the NetworkTables issue.

[davidthefat]

Wait, isn't this the reason why you debug and profile your code?

[connor.worley]

Originally Posted by davidthefat

Wait, isn't this the reason why you debug and profile your code?

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

[R.C.]

Originally Posted by connor.worley

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

Agreed, along with the limited time and constant updates.

-RC

[JB987]

Originally Posted by Basel A

While I don't think there's a strong case for replay or any change to the officially announced results of the 2012 FRC Championship, I do think it's worth noting that prior to their being knocked out, the Archimedes alliance was the only one suffering problems likely caused by FCA. If the interferer had an agenda, it seems that 1114, 2056, and 4334 was their target.

You will notice in the report that it also seems like 987 suffered the same problem in Final 1 and 2...

[Ross3098]

Originally Posted by JB987

You will see that we too were likely victims of the intentional"failed client authorization" debacle...

I cant help but believe that the Curie Curse had something to do with the Curie alliance in the finals....

[davidthefat]

Originally Posted by connor.worley

Yeah, but it's hard to hit edge cases, especially with so much hardware involved.

Well, I would disagree. From the language of the descriptions, it seems as if the problems would arise from general use. Sure, you'll definitely have to try hard to get certain conditions, but debugging and profiling your usual usage conditions is enough IMHO. Like the 100% CPU usage should have easily been caught.

edit: perhaps, it's just me. I like spending time with my robot.

[Nate Laverdure]

If I were the boss, I wouldn't have released this report on Friday the 13th.

[Basel A]

Originally Posted by JB987

You will notice in the report that it also seems like 987 suffered the same problem in Final 1 and 2...

True, but 987 didn't have those problems in the semifinals, suggesting that the Archimedes alliance was the higher-priority target. Of course, you're right, the fact that 987 suffered the same problem means that you were probably targeted as well. While it's regrettable that this happened to anyone, but it seems fairly clear that there was a pattern to the chaos which began with 1114, 2056, and 4334's issues.

[Jon Jack]

I think FIRST has handled the situation well so far. FIRST could have easily turned a blind eye and tried to sweep the problem under the rug. What would replaying the 2012 championship matches really solve? Nothing. In my opinion, giving the 12 teams automatic invites to the 2013 Championship as well as waiving their first play registration is enough consolation.

Originally Posted by Radical Pi

there's a large potential for unintentional FCA as well, which they didn't mention whether or not had been tested. Many teams configure their practice network at the shop with the same SSIDs that the field uses, and if they used a different security key, any phones/tablets/laptops that happened to be active during the games and had been connected to the network at home could have been attempting connections in the background. I actually caught our team's programming laptop trying this once during week 2

The issue isn't just trying to connect to the network, the issue is trying to connect to the network - while entering an incorrect WPA key:

During the Post-Championship field testing an attempt was made to connect to one of the team SSIDs set up on the field network, but the WPA key was entered incorrectly. This was observed to sever the communication
between the driver station and robot associated with that SSID. After further testing, a link between failed authentication attempts and a disruption of the communication between a driver station and robot was confirmed, though not every failed authentication attempt resulted in a communication disruption.

Lets hope that FIRST continues to learn from Einstein 2012 and makes changes to the control system to prevent this from happening at future events.

[apalrd]

There were several things I got out of this paper, especially as an engineer working on engine controllers:

-The Smart Dashboard had a bug which was exploited which caused a deadlock. While all software has bugs, it should also be tolerant of failure, meaning the rest of the system should have been designed to operate (possibly in limited quantity).

-The Smart Dashboard was mentioned numerous times relating to increased network load, especially the funny 1-byte packets.

-The VxWorks operating system handling of the packet buffer seems exceedingly poor. Many other forms of communication (e.g. some CAN stacks) dump old packets with the same ID when they are added to the buffer, this seems like the right move (at least on UDP).

-The boot time of the cRio was mentioned to be 24s minimum.
--I am currently working with an engine controller that can reboot the application software fast enough to not stall the (Diesel) engine while it is running.

-The nature of 802.11 makes it a poor choice for this kind of wireless communication.

I will not comment on anything else.

[Hjelstrom]

Originally Posted by davidthefat

Yes, I admit, I have not read carefully enough. Still, that's a shame IMHO. A deadlock anywhere is not good.

David, the deadlock is within the WPILibrary and we have been helping them resolve it. In fact, just today we were at the shop with our programmer Brandon as he finished up test programs for FIRST to use to reproduce two bugs we encountered in the smartdashboard support code this season.

These bugs did not affect our robot on Einstein or in any match all year because we avoided them. Working towards getting them fixed for next year is just a side benefit of the New Hampshire meeting.

[Alexa Stott]

Originally Posted by davidthefat

Well, I would disagree. From the language of the descriptions, it seems as if the problems would arise from general use. Sure, you'll definitely have to try hard to get certain conditions, but debugging and profiling your usual usage conditions is enough IMHO. Like the 100% CPU usage should have easily been caught.

edit: perhaps, it's just me. I like spending time with my robot.

If you read the document carefully, it states:

While this may have impacted the performance of the cRIO, it is unlikely that it was the source of 1114’s command response failures on Einstein.

[...]

No issues were found during these tests that could explain the connection problems seen on Einstein by 1114.

Later on, it states that the likely cause of 1114's issues was FCA.

Also:

It is unlikely that the code issues found on 4334’s robot relating to the 100% cRIO CPU usage would be able to cause a complete command response failure for the duration experienced in the initial playing of Semifinal 2-1.

The 100% CPU usage was not found to be the cause of their issues so debugging that issue probably would not have helped (much).

I really don't think many of the issues described in the document can be attributed to teams not debugging carefully enough.

[jason701802]

Originally Posted by Jon Jack

I think FIRST has handled the situation well so far. FIRST could have easily turned a blind eye and tried to sweep the problem under the rug.

You mean like they have been doing since they introduced this system? This report is several years too late, and even now it fails to address many issues. It is a shame that this report does nothing to address the issues at have been occurring at regionals since the introduction of this system.

[connor.worley]

Originally Posted by jason701802

You mean like they have been doing since they introduced this system? This report is several years too late, and even now it fails to address many issues. It is a shame that this report does nothing to address the issues at have been occurring at regionals since the introduction of this system.

Issues in years prior may have had nothing to do with the deauth attack- only one radio and one AP that they tested were vulnerable. Older radios may not be vulnerable. I don't know if the AP has been changed.