[FRC Blog] Einstein Report Released

[Racer26]

Originally Posted by JamesTerm

Now then when I observed the outcome of Archimedes eliminations... I noticed the blue wins as the #2 seed (blue) wins the division. Yeah... may not be significant, but noticeable to me.

This is simply an artifact of the very high level play at Championship, where the top several (in this case 2) alliances are capable of winning.

Originally Posted by JamesTerm

However, the FRC report itself does show a significant amount of failures for red alliance teams. Also, there was one match in particular that we played where we observed 2 simultaneous red alliance failures that got me thinking about this during the rest of the competition.

The report also says that it believes the bulk of the communications issues experienced during Einstein (other than 118's code/custom electronics issue) are believed to be the result of the Failed Client Authentication exploit. Einstein shouldn't show the red alliance bias that other elimination rounds do. If the targeted teams happened to be red, that's not statistically significant.

Also, in answer to Greg McKaskle's earlier post about some robots DITW with a flashing RSL not being the result of FCA: Here's a likely example. GTRWest (ON2)'s Finals match 2, 1114 dies partway through, with a flashing RSL. It was a Wk5 event. The vulnerability existed then. Probably one of the non-FCA reasons.

The first shot of them dead on camera is when the match clock reads 87 seconds remaining. Immediately before this shot, the red alliance station is shown, their red light lit solid. Match clock 54, the red alliance station is shown again, solid light. Based on my understanding of the failure mode of FCA, a solid alliance wall light is proof of something else being at fault. At match clock 51 of Einstein SF2.2, 2056's alliance wall light is seen flashing.

Knowing how the FCA bug operates, its easy to see how it could be caused accidentally, and I wonder just what percentage of the post-Wk4 comms issues could be attributed to it, even if the person operating the failed client didn't intend or even realize the consequences of their actions (or the automatic actions of their Wifi devices).

[Gregor]

Originally Posted by Ekcrbe

Each team affected by this has a right to be angry at the person who perpetrated the situation. But if four years later the team comes forward or is exposed, and those teams hold a grudge and still can't forgive the team the individual was connected to, then those teams have a GP issue. I would hope that stigma wouldn't last too long that far down the road.

Do they really have a GP issue? If someone killed another person and then waited 4 years to confess, would it be wrong for the family members to hold a grudge (an extreme case but you get my point)? As Tyler pointed out here, no one but the twelve Einstein teams know what it is like to be cheated out of a fair shot at the gold. If I were them, I'm sure I would still be angry 4 years from now. Heck, I'm not them and I'll probably still be angry. This post gives a very personal view into what these teams had to go through this season. Why should they forgive this individual? I think there would be a problem if these teams were not still angry after some time has passed (say 4 years?). I have no problem with teams holding a grudge after putting in hundreds of hours, dollars, and heartache into their teams, and that is only for this season alone. Multiply by how many years each team member has been involved with their team, and you get the magic number of how angry each and every one of them should be should be.

Originally Posted by stevend1994

Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons?

Thats what I read it as.

For personal reasons, this individual opted to resign.

Granted, "personal" reasons could mean anything from his/her dog dying to his/her overwhelming guilt for this inexcusable act of interference.

[apalrd]

Originally Posted by 1075guy

Knowing how the FCA bug operates, its easy to see how it could be caused accidentally, and I wonder just what percentage of the post-Wk4 comms issues could be attributed to it, even if the person operating the failed client didn't intend or even realize the consequences of their actions (or the automatic actions of their Wifi devices).

I wonder how many times the cause was the team's own laptop (e.g. a programming laptop left on the robot cart). I'm currently very glad we don't use 802.11 directly to our laptops at home (we still use that Linksys AP from the 2009 kit), thinking about this.

[Greg McKaskle]

1. Why was the there a significant amount of failures for just the red alliance teams?

Let's look for a mechanism by which the alliance color could influence the failure.

Is it actually the color of the bumpers? Will putting red bumpers on a robot make it fail, or will blue bumpers fix it? Of course not.

What about the wifi? It turns out that all traffic is sent out on the same wifi frequency and channel. Red and Blue robot traffic is literally chopped into small packets and transmitted one after the other in a big data stream. RF spectral noise cannot bias red or blue or any particular robot. Additional APs using the same channel will not bias based on color or robot.

The data is sent out over the same antennae. The AP has six antennae, but three are tuned for 2.4 GHz and three for 5 GHz. The three antennae are used in MIMO fashion to modulate the bits of each packet. No bias there. The packets are the same size, and only difference is that the control packets have either an ASCII R or B within an inner field. B is 0x42, R is 0x52. The adjacent field has an ASCII 1, 2, or 3 for the alliance station by the way. Team ID is used as the SSID name. This resolves to a unique BSSID which is one of the six MAC address of the router. The MAC is in the IP packet header and nothing about team number, color, or anything else is. No opportunity for bias that I see.

That leaves us with the physical cables that deliver the data from the DS to the wifi AP and the switches that merge and route them. I don't have an explanation for how they would bias.

Finally, the report gives details as to root cause. I don't see failures following colors or stations. I mostly see things following robots.

Please explain the mechanism or let this one die. It is a small sample size and robots in elims do not swap bumpers.

2. Special interest in how much vision tracking contributes to network traffic, as we (beta team)... were concerned of overwhelming traffic from this for teams that wanted to process vision via driver station.

Vision processing that stays on the robot has no impact. Vision streams that are sent to the dashboard via TCP can be estimated and the utilization of the particular teams on Einstein was measured and compared to the estimate. A color 640x480 image stream has 921,600 bytes of pixel data per image. with compression, it is likely 30kB per image frame. If the team sets the framerate to the highest of 30fps, that gives less than a MB per second or about 10Mbits. For six robots that is a respectable 60Mbits for video alone. The rest of the field data takes about 6.5Mbits. True, some robots could have multiple cameras, and they could set compression such that images were three or four times larger, but most teams will not have 30fps and 640x480 and low compression. The Einstein fields were usually measured to use 25Mbits. I wouldn't want to use g networks, and n needs a clean channel to achieve lots of throughput, but the channel bandwidth is not taxed by vision cameras ... yet. mp4 compression will work fine if both camera and laptop have codecs. The cRIO doesn't.

I don't want to talk about the political stuff but just want to throw some general things out there... 1. Is there a scapegoat political agenda going on? 2. No one can keep a secret as all will be revealed to those who want to know about it... it is just a matter of time. Why? friends tell friends, and those friends tell friends... and well you get the picture... just like FB itself.

1. How about no. FIRST staff, experts, and the 12 Einstein teams in attendance were each asked if they approve of the report findings. All agreed. Explain why they would all have the same agenda?
2. Well if it is true for FaceBook, ...

Greg McKaskle

[Ekcrbe]

Originally Posted by Gregor

Do they really have a GP issue? If someone killed another person and then waited 4 years to confess, would it be wrong for the family members to hold a grudge (an extreme case but you get my point)? As Tyler pointed out here, no one but the twelve Einstein teams know what it is like to be cheated out of a fair shot at the gold. If I were them, I'm sure I would still be angry 4 years from now. Heck, I'm not them and I'll probably still be angry. This post gives a very personal view into what these teams had to go through this season. Why should they forgive this individual? I think there would be a problem if these teams were not still angry after some time has passed (say 4 years?). I have no problem with teams holding a grudge after putting in hundreds of hours, dollars, and heartache into their teams, and that is only for this season alone. Multiply by how many years each team member has been involved with their team, and you get the magic number of how angry each and every one of them should be should be.

No, no, no. I'm sorry for any ambiguity in my previous post. I meant it would be wrong for the victims to be mad at the associated team, not the individual him/herself. After four years of time to numb the pain slightly, I would expect reasonable people, like FIRSTers, to have cleared their vision and then be able to see that the associated team itself is not (we hope) the bad guy. The rest of that team could be, in a roundabout way, like victims themselves, because of the horrible publicity potentially brought down upon them by the community at large.
I hope that hailstorm doesn't come to them.

To what you did say, of course they will be mad, and justified in being so.

[Gregor]

Originally Posted by Ekcrbe

No, no, no. I'm sorry for any ambiguity in my previous post. I meant it would be wrong for the victims to be mad at the associated team, not the individual him/herself. After four years of time to numb the pain slightly, I would expect reasonable people, like FIRSTers, to have cleared their vision and then be able to see that the associated team itself is not (we hope) the bad guy. The rest of that team could be, in a roundabout way, like victims themselves, because of the horrible publicity potentially brought down upon them by the community at large.
I hope that hailstorm doesn't come to them.

To what you did say, of course they will be mad, and justified in being so.

When the individual's name and assoisted team are finally discovered, I would like to think that the FIRST community wouldn't hold the team to blame. I couldn't imagine it if someone from my team did something like this. That would already be agonizing for the team in itself. To then be know as the team who ruined Einstein 2012 will be a sad, yet albeit inevitable add on.

[Steven Donow]

Originally Posted by Gregor

what I read it as.

Granted, "personal" reasons could mean anything from his/her dog dying to his/her overwhelming guilt for this inexcusable act of interference.

Well, I was moreso saying that that's what they'd tell their team; "I'm leaving for random understandable personal reason" when it's really "I'm banned from FIRST events". With their team having NO knowledge whatsoever of what he/she did


And does anyone else think it might be beneficial(or in some ways detrimental) to have two topics of discussion for thisne political, one technical. Or would that just draw too much attention to the political stuff/become a battleground?

[Ekcrbe]

Originally Posted by Gregor

To then be know as the team who ruined Einstein 2012 will be a sad, yet albeit inevitable add on.

I challenge the FIRST community to be above this and avoid putting this stigma on an unfortunate team and consider everyone in this team's influence innocent until proven guilty. There's no guarantee the perpetrator acted alone, but it would be even sadder and stranger to find a cast of co-conspirators. I doubt this will be the case.

[Akash Rastogi]

Did anyone from 1717 get to formally investigate with FIRST the cause of their issues during division eliminations?

Seems logical to look into their case as well as other communication issues during eliminations at Champs, albeit, probably without inviting all teams to Manchester it might be difficult. At least it would give these other teams piece of mind as well knowing that it is not solely some issue with code/electronics that caused their issues during elims.

[David Brinza]

Originally Posted by stevend1994

<snip> If the team name never is revealed other than mumblings at regionals (ie. "Yeah I hear it was that mentor for team XXXX"), then that is even better.

Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons?

I think we're all shocked by what occurred in St. Louis. This is NOT FIRST.

Suppose the perpetrator leaves the team because of graduation? The "individual" was not identified in the FIRST Einstein Investigation Report as being a mentor nor student. Wouldn't the FIRST community would be more forgiving if this was more like a teenage "stunt" rather than an attack on sportsmanship and GP by a team leader? If a mentor did this, the team involved likely would suffer irreparable damage to their reputation.

[Todd]

Good work on FIRST and all volunteers and afflicted team's parts with their swift and thorough analysis.

Good work on catching the issues with smart dashboard.

Good work on 118's part in identifying their own issue with their gyro reset loop.

Good work on 118's and volunteers parts again for working to identify the issue with their vxworks network buffer being overrun following the cessation of normal crio code activity.

Good work on all parts reminding us that its probably best that our cRios never run at 100% utilization, and that its perfectly plausible to not hit 100% doing all sorts of complicated logic, whether you're running c++, java, or labview.

Good work on deciding to limit team bandwidth moving into the future, our applications are only going to get greedier with time and we need to have a cap.

Good work spotting all wiring problems that were in fact present with several of the playoff teams.

And good work and god speed in all future efforts in securing the field against any external interference, be it intentional or not. The only secure network is one that's disconnected, but we can all (and I'm sure will all) do our best to improve the quality of experience for all FIRST teams.

[JamesTerm]

Originally Posted by Greg McKaskle

Let's look for a mechanism by which the alliance color could influence the failure.
...
Please explain the mechanism or let this one die. It is a small sample size and robots in elims do not swap bumpers.
Greg McKaskle

Thanks for the detailed analysis on this. I cannot explain the mechanism as I am not an expert in networking. I just observed one side having a signifcant amount of failures in the report. Since no-one acknowledged this in the report I wanted to... on the long shot chance that maybe (just maybe)... like tracing wires to a defected part... there could be something there.

Originally Posted by Greg McKaskle

The Einstein fields were usually measured to use 25Mbits. I wouldn't want to use g networks, and n needs a clean channel to achieve lots of throughput, but the channel bandwidth is not taxed by vision cameras ... yet. mp4 compression will work fine if both camera and laptop have codecs. The cRIO doesn't.
Greg McKaskle

Thanks very much for these numbers! Looking forward to seeing how they distrubute the network capping in future documents. One thing I'll want to determine is if the mp4 offerred by say the 1101, is if it actually uses p and b frames... or if it is i frame only. I know AVCHD like canon vixia does a great job with compression by using delta frames accordingly.

Originally Posted by Greg McKaskle

1. How about no. FIRST staff, experts, and the 12 Einstein teams in attendance were each asked if they approve of the report findings. All agreed. Explain why they would all have the same agenda?
Greg McKaskle

I don't think so either.

[Al Skierkiewicz]

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

[IndySam]

Originally Posted by Al Skierkiewicz

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

/end discussion

[JamesTerm]

There is one more thing I wish to share which was a symptom 100% reproducible where during the switch from autonomous to teleop showed no connection and we continued to have lost connection until near the end of the match.

This happened at our first match last year... Unfortunately I do not know the full reason as to why, but it had to do with how much work was done cleaning up objects that ran in autonomous and starting similar ones up again in teleop. The good news is that we could reproduce this problem using the practice button (So yes there is good reason to test with it). We had to disable autonomous for that competition to avoid this symptom.

I should add using the practice button is a great test, but it could give a false positive. It may be that the instantiation (or cleanup) is just on the threshold and with any more network time delay could throw it over the edge. I just want to throw a word of caution to all programmers to not make the same mistake we did. Be aware of how much work occurs between ending autonomous and starting teleop(). We never found the root cause of this, so I do not know if the vulnerability still exists.

What I can tell you is that we instantiate everything on powerup, and do the minimal amount of work to transition from autonomous to teleop.

In regards to the current subject at hand... I just want to add this as a diagnostic check if a similar symptom occurs or had occurred.