Re: [FRC Blog] Einstein Report Released
 

I have to think he's talking about elimination matches, where there *should* be a decided red bias.

Re: [FRC Blog] Einstein Report Released
 

Yup. There should be a fairly strong Red bias in Elims, especially so in the QFs.

Side note of interest, at ON2 (GTR-West) this year, 2056 played their first Elimination matches at a regional on the blue alliance. Every other regional, they've won from the #1 seeded alliance, who never plays with blue bumpers.

Re: [FRC Blog] Einstein Report Released
 

To those that believe that the team of the INDIVIDUAL should hesitate to come forward, let's think about our culture for a minute.

In my experience, FIRST is about accountability and responsibility. We have an accountability to others, both for our good decisions and our bad. We have a responsibility to make the world a better place than when we arrived in it. This means we hold ourselves accountable for our mistakes and celebrate our successes.

On my team I've made mistakes. We all do. We don't hide them from others. We learn from them. We discuss them. We are accountable to ourselves but also those within our community. I know for a fact at certain points during my time on the team many mentors would have called me an arrogant jerk, and they'd be right. But the culture of our team and of FIRST allowed me to accept my mistakes and grow. Now those same mentors have hired me at their companies and have given me great recommendations because I made myself accountable for my actions and learned from them.

Now about this individual. I do believe that they are beyond being forgiven. Their actions have a malicious intent that I believe is clearly illustrated by the actions of FIRST in dealing with them. FIRST doesn't reserve that kind of action for an accident. The individual's actions were deeply hurtful to everyone who has even been touched by FIRST, and I believe that they don't really care given,

1.Their actions
2. A lack of an apology thus far

That said they will only be anonymous for so long. Eventually the community will have a name and a number to go with it.

My opinion is that team should come forward. Help the community heal and acknowledge that a team member turned out to be a different person than you thought. Don't help protect this person for your own anonymity. That can only last so long. If you truly did nothing wrong no one who matters and has half a brain would hold you accountable for something that you are not to blame for. Don't protect someone who chose to make a decision that terrible. Address the situation in a manner befitting the quality of the community we are all a part of.

The teams targeted are some of the best. Ever. They have worked to excel to a point that most of us will never reach. They should be celebrated for reaching that level and helping inspire others to do so. I honestly think they deserve commendation for simply maintaining a positive attitude despite the malicious actions of some idiots over the years (and yes they are idiots, no apologies). They deserve an apology and an explanation. Anyone in their place does.

In short, to everyone. Be accountable. Acknowledge success and mistakes, and grow from them.

Just my thoughts on a bad situation that had a fortune to be dealt with by some amazing people.

Re: [FRC Blog] Einstein Report Released
 

Attached Excel2000 spreadsheet is from the FRC Twitter data feed.

The usual caveats about the FRC Twitter data feed apply.

Re: [FRC Blog] Einstein Report Released
 

Each team affected by this has a right to be angry at the person who perpetrated the situation. But if four years later the team comes forward or is exposed, and those teams hold a grudge and still can't forgive the team the individual was connected to, then those teams have a GP issue. I would hope that stigma wouldn't last too long that far down the road.

Everything else is spot on.

Re: [FRC Blog] Einstein Report Released
 

Cool you and qzrrbz know what I'm getting at... I noticed a strong Red bias in my observations of regional elimination matches (Dad1279, qzrrbz is right I was talking about eliminations only) and I believe it to due to how the higher seed is on the red alliance. Now then when I observed the outcome of Archimedes eliminations... I noticed the blue wins as the #2 seed (blue) wins the division. Yeah... may not be significant, but noticeable to me.

Ether: Thanks for this data... it confirms what 1075guy and I have observed in regards to the red bias in elims. From this global scope xls document everything is probably as we'd expect. However, the FRC report itself does show a significant amount of failures for red alliance teams. Also, there was one match in particular that we played where we observed 2 simultaneous red alliance failures that got me thinking about this during the rest of the competition.

Re: [FRC Blog] Einstein Report Released
 

I was hoping for an answer or opinion, not another question. ;)
It should be self-explanitory... if not then please just disregard.

Re: [FRC Blog] Einstein Report Released
 

This is what I mean. The situation in which the team formerly associated with the individual steps forward and makes a statement will create a much better atmosphere for themselves than the situation where the community eventually finds out who this person is and publishes their name and association. As much as I think that illustrates the bad parts of the internet, it's more than likely going to happen because this IS such an emotionally charged issue for so many.

If you were that team, wouldn't you want to admit that, like Alex said, this person had a different character than you thought... rather than get called out on a public forum before you get your chance to speak for yourself?

I don't approve of a witch-hunt for the person involved, but I do believe the team should get a chance to step forward before those with the torches and pitchforks get them first.

Re: [FRC Blog] Einstein Report Released
 

I disagree...I don't see how a team's statement would be any different announcing it themselves or after the "witch-hunt" finishes. I, for one, would feel no different feelings for the team other than, if they announced it themselves, saying, 'that's bold', and then moving on with the same 'not the team's fault, etc...' feelings. But as EricH greatly said earlier, that team will forever have their name associated with this, whether that's the right thing or not; it's what will happen.

The longer the secret is kept, the better. If the team name never is revealed other than mumblings at regionals(ie. "Yeah I hear it was that mentor for team XXXX"), then that is even better.


Also, has anyone considered the possibility that the team themselves don't even know about it, and that the perpetrator is passing it off as leaving for other reasons? Part of me wants to view that as a case, but part of me has reasons why that wouldn't work (how would FIRST get into contact with the individual other than through the main team contact? unless that person kept it a complete secret/between them and the perpetrator).

Re: [FRC Blog] Einstein Report Released
 

I'm happy for the FMS Whitepaper.

Re: [FRC Blog] Einstein Report Released
 

This is simply an artifact of the very high level play at Championship, where the top several (in this case 2) alliances are capable of winning.

The report also says that it believes the bulk of the communications issues experienced during Einstein (other than 118's code/custom electronics issue) are believed to be the result of the Failed Client Authentication exploit. Einstein shouldn't show the red alliance bias that other elimination rounds do. If the targeted teams happened to be red, that's not statistically significant.

Also, in answer to Greg McKaskle's earlier post about some robots DITW with a flashing RSL not being the result of FCA: Here's a likely example. GTRWest (ON2)'s Finals match 2, 1114 dies partway through, with a flashing RSL. It was a Wk5 event. The vulnerability existed then. Probably one of the non-FCA reasons.

The first shot of them dead on camera is when the match clock reads 87 seconds remaining. Immediately before this shot, the red alliance station is shown, their red light lit solid. Match clock 54, the red alliance station is shown again, solid light. Based on my understanding of the failure mode of FCA, a solid alliance wall light is proof of something else being at fault. At match clock 51 of Einstein SF2.2, 2056's alliance wall light is seen flashing.

Knowing how the FCA bug operates, its easy to see how it could be caused accidentally, and I wonder just what percentage of the post-Wk4 comms issues could be attributed to it, even if the person operating the failed client didn't intend or even realize the consequences of their actions (or the automatic actions of their Wifi devices).

Re: [FRC Blog] Einstein Report Released
 

Do they really have a GP issue? If someone killed another person and then waited 4 years to confess, would it be wrong for the family members to hold a grudge (an extreme case but you get my point)? As Tyler pointed out here, no one but the twelve Einstein teams know what it is like to be cheated out of a fair shot at the gold. If I were them, I'm sure I would still be angry 4 years from now. Heck, I'm not them and I'll probably still be angry. This post gives a very personal view into what these teams had to go through this season. Why should they forgive this individual? I think there would be a problem if these teams were not still angry after some time has passed (say 4 years?). I have no problem with teams holding a grudge after putting in hundreds of hours, dollars, and heartache into their teams, and that is only for this season alone. Multiply by how many years each team member has been involved with their team, and you get the magic number of how angry each and every one of them should be should be.


Thats what I read it as.
Granted, "personal" reasons could mean anything from his/her dog dying to his/her overwhelming guilt for this inexcusable act of interference.

Re: [FRC Blog] Einstein Report Released
 

I wonder how many times the cause was the team's own laptop (e.g. a programming laptop left on the robot cart). I'm currently very glad we don't use 802.11 directly to our laptops at home (we still use that Linksys AP from the 2009 kit), thinking about this.

Re: [FRC Blog] Einstein Report Released
 

Let's look for a mechanism by which the alliance color could influence the failure.

Is it actually the color of the bumpers? Will putting red bumpers on a robot make it fail, or will blue bumpers fix it? Of course not.

What about the wifi? It turns out that all traffic is sent out on the same wifi frequency and channel. Red and Blue robot traffic is literally chopped into small packets and transmitted one after the other in a big data stream. RF spectral noise cannot bias red or blue or any particular robot. Additional APs using the same channel will not bias based on color or robot.

The data is sent out over the same antennae. The AP has six antennae, but three are tuned for 2.4 GHz and three for 5 GHz. The three antennae are used in MIMO fashion to modulate the bits of each packet. No bias there. The packets are the same size, and only difference is that the control packets have either an ASCII R or B within an inner field. B is 0x42, R is 0x52. The adjacent field has an ASCII 1, 2, or 3 for the alliance station by the way. Team ID is used as the SSID name. This resolves to a unique BSSID which is one of the six MAC address of the router. The MAC is in the IP packet header and nothing about team number, color, or anything else is. No opportunity for bias that I see.

That leaves us with the physical cables that deliver the data from the DS to the wifi AP and the switches that merge and route them. I don't have an explanation for how they would bias.

Finally, the report gives details as to root cause. I don't see failures following colors or stations. I mostly see things following robots.

Please explain the mechanism or let this one die. It is a small sample size and robots in elims do not swap bumpers.

Vision processing that stays on the robot has no impact. Vision streams that are sent to the dashboard via TCP can be estimated and the utilization of the particular teams on Einstein was measured and compared to the estimate. A color 640x480 image stream has 921,600 bytes of pixel data per image. with compression, it is likely 30kB per image frame. If the team sets the framerate to the highest of 30fps, that gives less than a MB per second or about 10Mbits. For six robots that is a respectable 60Mbits for video alone. The rest of the field data takes about 6.5Mbits. True, some robots could have multiple cameras, and they could set compression such that images were three or four times larger, but most teams will not have 30fps and 640x480 and low compression. The Einstein fields were usually measured to use 25Mbits. I wouldn't want to use g networks, and n needs a clean channel to achieve lots of throughput, but the channel bandwidth is not taxed by vision cameras ... yet. mp4 compression will work fine if both camera and laptop have codecs. The cRIO doesn't.

1. How about no. FIRST staff, experts, and the 12 Einstein teams in attendance were each asked if they approve of the report findings. All agreed. Explain why they would all have the same agenda?
2. Well if it is true for FaceBook, ...

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

No, no, no. I'm sorry for any ambiguity in my previous post. I meant it would be wrong for the victims to be mad at the associated team, not the individual him/herself. After four years of time to numb the pain slightly, I would expect reasonable people, like FIRSTers, to have cleared their vision and then be able to see that the associated team itself is not (we hope) the bad guy. The rest of that team could be, in a roundabout way, like victims themselves, because of the horrible publicity potentially brought down upon them by the community at large.
I hope that hailstorm doesn't come to them.

To what you did say, of course they will be mad, and justified in being so.

Re: [FRC Blog] Einstein Report Released
 

When the individual's name and assoisted team are finally discovered, I would like to think that the FIRST community wouldn't hold the team to blame. I couldn't imagine it if someone from my team did something like this. That would already be agonizing for the team in itself. To then be know as the team who ruined Einstein 2012 will be a sad, yet albeit inevitable add on.

Re: [FRC Blog] Einstein Report Released
 

Well, I was moreso saying that that's what they'd tell their team; "I'm leaving for random understandable personal reason" when it's really "I'm banned from FIRST events". With their team having NO knowledge whatsoever of what he/she did


And does anyone else think it might be beneficial(or in some ways detrimental) to have two topics of discussion for this:one political, one technical. Or would that just draw too much attention to the political stuff/become a battleground?

Re: [FRC Blog] Einstein Report Released
 

I challenge the FIRST community to be above this and avoid putting this stigma on an unfortunate team and consider everyone in this team's influence innocent until proven guilty. There's no guarantee the perpetrator acted alone, but it would be even sadder and stranger to find a cast of co-conspirators. I doubt this will be the case.

Re: [FRC Blog] Einstein Report Released
 

Did anyone from 1717 get to formally investigate with FIRST the cause of their issues during division eliminations?

Seems logical to look into their case as well as other communication issues during eliminations at Champs, albeit, probably without inviting all teams to Manchester it might be difficult. At least it would give these other teams piece of mind as well knowing that it is not solely some issue with code/electronics that caused their issues during elims.

Re: [FRC Blog] Einstein Report Released
 

I think we're all shocked by what occurred in St. Louis. This is NOT FIRST.

Suppose the perpetrator leaves the team because of graduation? The "individual" was not identified in the FIRST Einstein Investigation Report as being a mentor nor student. Wouldn't the FIRST community would be more forgiving if this was more like a teenage "stunt" rather than an attack on sportsmanship and GP by a team leader? If a mentor did this, the team involved likely would suffer irreparable damage to their reputation.

Re: [FRC Blog] Einstein Report Released
 

Good work on FIRST and all volunteers and afflicted team's parts with their swift and thorough analysis.

Good work on catching the issues with smart dashboard.

Good work on 118's part in identifying their own issue with their gyro reset loop.

Good work on 118's and volunteers parts again for working to identify the issue with their vxworks network buffer being overrun following the cessation of normal crio code activity.

Good work on all parts reminding us that its probably best that our cRios never run at 100% utilization, and that its perfectly plausible to not hit 100% doing all sorts of complicated logic, whether you're running c++, java, or labview.

Good work on deciding to limit team bandwidth moving into the future, our applications are only going to get greedier with time and we need to have a cap.

Good work spotting all wiring problems that were in fact present with several of the playoff teams.

And good work and god speed in all future efforts in securing the field against any external interference, be it intentional or not. The only secure network is one that's disconnected, but we can all (and I'm sure will all) do our best to improve the quality of experience for all FIRST teams.

Re: [FRC Blog] Einstein Report Released
 

Thanks for the detailed analysis on this. I cannot explain the mechanism as I am not an expert in networking. I just observed one side having a signifcant amount of failures in the report. Since no-one acknowledged this in the report I wanted to... on the long shot chance that maybe (just maybe)... like tracing wires to a defected part... there could be something there.


Thanks very much for these numbers! Looking forward to seeing how they distrubute the network capping in future documents. One thing I'll want to determine is if the mp4 offerred by say the 1101, is if it actually uses p and b frames... or if it is i frame only. I know AVCHD like canon vixia does a great job with compression by using delta frames accordingly.


I don't think so either.

Re: [FRC Blog] Einstein Report Released
 

Everyone,
It is my firm belief that the individual involved acted alone, without the knowledge of the team. In fact the team was cooperative in the investigation. A witch hunt to determine the team involved serves no purpose. Further I do not believe the person involved in the attack did so to target a specific team and prevent them from winning. The choice of which team to attack seemed merely a means to an end to prove that a robot was vulnerable.
The sentence for the individual as spelled out in Jon's letter was harsh but just, as it should be. Should someone else, student or mentor, discover an issue in the future that compromises the competition, I hope that this sentence will dissuade them from demonstrating the issue during match play.
I hold no ill will against this team and will gladly play with them in the future. In my opinion the mentors demonstrated GP once they were aware of the issue. I doubt students on the team were aware of the situation at the time. Any further action will only serve to harm the team, the students they serve, and the community as a whole. I wish them well in the future.

Re: [FRC Blog] Einstein Report Released
 

/end discussion

Re: [FRC Blog] Einstein Report Released
 

There is one more thing I wish to share which was a symptom 100% reproducible where during the switch from autonomous to teleop showed no connection and we continued to have lost connection until near the end of the match.

This happened at our first match last year... Unfortunately I do not know the full reason as to why, but it had to do with how much work was done cleaning up objects that ran in autonomous and starting similar ones up again in teleop. The good news is that we could reproduce this problem using the practice button (So yes there is good reason to test with it). We had to disable autonomous for that competition to avoid this symptom.

I should add using the practice button is a great test, but it could give a false positive. It may be that the instantiation (or cleanup) is just on the threshold and with any more network time delay could throw it over the edge. I just want to throw a word of caution to all programmers to not make the same mistake we did. Be aware of how much work occurs between ending autonomous and starting teleop(). We never found the root cause of this, so I do not know if the vulnerability still exists.

What I can tell you is that we instantiate everything on powerup, and do the minimal amount of work to transition from autonomous to teleop.

In regards to the current subject at hand... I just want to add this as a diagnostic check if a similar symptom occurs or had occurred.

Re: [FRC Blog] Einstein Report Released
 

I've watched this for the last two days and have been trying to find a way to write what Al did.

Lets look to fix the control system problems. Remind roboteers and mentors that a 125LB robot moving at 9FPS slamming into another robot creates a lot of force so connectors, cables and COTS electronics may be damaged. Put as much design effort into our electrics and electronics as we do for our mechanics. Remind designers that CPU cycles and network capacity is not infinite, less is always more. Make every CPU cycle and network message count.

Lets pull together as a community on these topics. All teams now and in the future will benefit from the shared knowledge of how to do these things.

But as Al and IndySam said:

Add me to that list too. Nobody benefits from retribution.

Re: [FRC Blog] Einstein Report Released
 

I would like to know the team simply because it seems they could use a hug.

Re: [FRC Blog] Einstein Report Released
 

There are ways to improve the time to troubleshoot power issues on the robot and they should be explored. Documentation is a fine thing but at some point we have lots of documentation now and all too often it's missed. Writing documents is fine to offload responsibility for risk to those that should be reading those documents. However, as Einstein demonstrates shifting responsibility does little for the reputation of FIRST generally. The people in the stands are there to see the show and robots not moving is what you expect at a museum.

There are proper ways and improper ways to conduct investigations if you know you have a real issue. I should hope the message of this report is tempered to consider that not everyone who takes issues with the status quo is going to operate with reckless disregard for policy and process. I should also hope that this report makes it clear to FIRST that stonewalling channels can lead to negative outcomes. Just a point as I have often found that it's a serious problem to get the attention of those that have the power to command resources to investigate within the organization. That alone can never justify the risk of this situation but I can see how the factor contributes.

From my last post it's clear that ranking or de-ranking Team 25 (based only on the unique information I have as I have their original A version router) couldn't have been the practical goal. Under the circumstances any effect on who won was probably as stated by Al and this report merely secondary to the person's goal to demonstrate the cause and effect of their ability to deny service to robots (regardless of whom was effected which in many ways made their point useless anyway).

Given the only person who has been confirmed to cause any issue was only doing something that anyone could have and probably did stumble over (hence some of the earlier failures). I return to the same point as the others. This was a bad decision on their part and they have paid their price according to what FIRST deemed acceptable. FIRST is the most effected. FIRST lost money, reputation, resources, and opportunity on a national stage as the direct result.

As to the idea presented elsewhere that no team can understand how these teams felt. Let me point out that frequently teams ranking towards the top are effected by the ruling of the refs on the behavior of other teams on the fields during matches. I have personally been involved with a team that felt they were improperly impacted by the actions of other teams and people on the field during play. While this is more unusual than normal it's really quite similar. These are all fantastic teams. I feel badly for the seniors on these teams who will soon no longer be able to hold student roles, but beyond that these teams are not magically great. They have important recognized qualities that will carry them past this and we hope to future recognition.

Let no one mistake that I still think the smart thing for FIRST to do is legally bind these people to silence and strongly discourage any attempt to discover who these people are. We are a large organization. I worry not about the majority of FIRST and how it will behave but I worry that somewhere in FIRST there is a small number of people that will play hero if they find out and only compound the misdeed.

Let us focus on the positive which is that FIRST can move forward into the future hopefully better for this.

Re: [FRC Blog] Einstein Report Released
 

Well no, but it could influence the drivers.
http://www.sciencentral.com/video/20...pic-advantage/

:cool:
/tangent

Re: [FRC Blog] Einstein Report Released
 

This is probably why i am more scared of 1114 than 2056. lol (I love them both though :) )

Re: [FRC Blog] Einstein Report Released
 

I wonder what is going through their minds when they face each other! :P

Re: [FRC Blog] Einstein Report Released
 

Especially if 1114 is in blue and 2056 is in red...

Re: [FRC Blog] Einstein Report Released
 

I believe he's referring to the colour of the teams individually, and not when there on the field (Bumpers).

Re: [FRC Blog] Einstein Report Released
 

That's the point! If their team colors oppose their bumper colors, who has the advantage?

Re: [FRC Blog] Einstein Report Released
 

I'm really proud of FIRST and its volunteers for putting such a thorough report together. This was a fascinating thread to read, and I appreciate everyone pretty much holding it together and providing a good discussion.

Although unrelated to the report, it did make me finish my recap of Championship. You can read it here.

Re: [FRC Blog] Einstein Report Released
 

Everyone else :D

Re: [FRC Blog] Einstein Report Released
 

DUNT DUNT DUHHHHHHH

lmfao

Re: [FRC Blog] Einstein Report Released
 

With the passage of time, things get clearer in my head but I don't know if I am happier about the situation.

First, for my own part, I apologize unreservedly to FIRST Canada. My early post was totally unfounded. I am sorry for implying that perhaps this was in any way connected to bad blood associated with the Greater Toronto East Regional. This was very unfair of me.

Second, as to naming the individual and the team, it is clear from the various eye witness accounts that a LOT of people know who this person was and what team they were associated with. KEEPING THIS A SECRET IS UNTENABLE. It will get out. Once it does, I suppose that there will be a lot of digging around looking for reasons why this information was kept secret... ...and I assure you that it is always possible for conspiracy theorists to weave together plausible narratives that will paint the individual, team and FIRST in a much worse light than if they had published the information themselves.

I BEG FIRST, the associated team and even the individual to publish what they know. While it is painful in the short run, it is going to be better to get this out in the light of day now.

While they are at it, they should probably address the rapid departure of Bill Miller. Even the most charitable observer cannot help but wonder if his departure was related to all of this. I am not saying it is or it isn't but I think it is in FIRST's best interested to address this openly.

Calling 'em as I see 'em.

Joe J.

Re: [FRC Blog] Einstein Report Released
 

The best thing about forum discussions is that we don't need to be physically with the person (/ team / entity) to give them a cyber-hug.

So who's with Taylor and me, and wants to give the currently (and hopefully permanently) unknown team a group hug?? Come on, CD community, let's set an internet record!

Cyber-Group-Hug time! :)

Re: [FRC Blog] Einstein Report Released
 

Does that mean we're having another unofficial ChiefDelphi web hug?? :D :rolleyes:

Re: [FRC Blog] Einstein Report Released
 

Web-ier and hug-ier than ever before! But wait! There's more! Call within the next 10 minutes and we'll throw in an extra hug FREE! (just pay processing and handling)

Re: [FRC Blog] Einstein Report Released
 

Wow a web hug??

Re: [FRC Blog] Einstein Report Released
 

Virtual, of course. :)

I've waited until the report digested properly to comment.

The short version is: Good job everyone, that's a good report. Thank you. 1676 suffered lots of comm problems throughout the season, and I'm 98% convinced that our code is at fault.

Now for the hard part: Finding it...

Re: [FRC Blog] Einstein Report Released
 

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

I was standing by the field during Einstein with no team in the competition, only for the opportunity to be there with friends from other teams and try to give them support in the finals. I can honestly say I have never had such a range of emotions brought on by any series of events then these finals. I witnessed the discussions, I vented my frustrations, and I offered my sympathies and congratulations to my friends. One thing remains is that I have never been so angry at a robotics event, to the point that I felt physically ill after the event. ALL THIS WITHOUT MY TEAM COMPETING. My heart will always go out to my friends on the teams that will have to live with the feelings of being taken advantage of, and the thoughts of "what if" for the rest of their lives. Sure, the pain will lessen but you never forget these things.

Afterwards many people, including myself, second guessed how FIRST handled the situation and wished that things had been done differently regarding on the spot trouble shooting, but not having been in their shoes and knowing that hindsight is always 20/20, I have nothing to say on this aspect of the situation.

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure. In all honesty, part of me wishes that it had just been a random occurrence caused by almost anything else. The fact that someone knowingly sabotaged the matches is UNFATHOMABLE in the community which we have. Not only has this person impacted the lives of the teams he stole from, but has brought into question how trusting we can be with our community. I hope that this is an isolated incident, but sabotage or interference will now be in the minds of every competing team for both future and past events.

The one thing that is left still unresolved in this whole situation is a public statement from the individual and team involved.

The person who did this needs to stand up for his actions, not just to provide a sense of closure to the situation, but also to grow from his mistakes. One of the hardest things in life a person has to do is admit that they were wrong, and while the punishment has already been dealt out, it is a showing of personal maturity about the way the situation should be handled. When you make a mistake you must come clean for both the people impacted by your decisions, and to clear your conscious. In addition to a public apology I believe the right thing to do would be to reveal each instance that this exploit was used in order to clear the air and remove the doubt that has clouded the minds of every competitor at events from week 4 onward.

It is going to be a rough road if (and hopefully when) you do come forward, but you should sleep in the bed you make, and in the end I promise you that you will be better person for it.

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.



Greg

Re: [FRC Blog] Einstein Report Released
 

Greg, I really respect this post and your opinion.

Re: [FRC Blog] Einstein Report Released
 

+1

This last part of your post really stood out to me. Accountability is also a way for you to earn at least a little respect back from those you have wronged; some sort of decency is kept. The speculation and rumors that spread amongst the community may also sometimes breed animosity for the entire team, regardless of who took part in the wrong-doing. The longer this team/person holds out on some form of apology, the less and less respect, forgiveness, and sense of decency they should expect from the FIRST community and especially the teams they affected.

On another note- just read tonight that a team who will be attending IRI was able to fix bugs in their robot thanks to the Einstein report and I hope this is the case for many others, so +1 to all those who took part in the testing.

Re: [FRC Blog] Einstein Report Released
 

This, literally this.

It is healthy to admit your mistakes. No one is perfect, we have all done things wrong. It takes a big person to understand their mistakes and admit that they were wrong. Too many people in the world believe that they are perfect, or good enough and have it all figured out, and it's just not right or healthy.

Furthermore, in addition to what Greg said, I ask that if and when a public apology comes out, that the first community would be able to do an even harder thing: forgive the individual. Just as it is unhealthy to harbor your wrongdoing in secret, it is just as unhealthy to hold a grudge against those who have wronged you.

This individual is not the only one who needs to be prepared for when (hopefully when) the public apology comes out. Is each individual in the first community, and more specifically the 12 Einstein teams, ready for the apology? Are you ready to realize that what's done is done, to forgive the individual, and work to ensure that this does not happen again? I think each person invested in this community needs to ask themselves this question.

Although it is hard to forgive and it's natural to be angry, there is no benefit in holding the hatred in your heart.

~$0.02
Alex

Re: [FRC Blog] Einstein Report Released
 

"You understand we're making a conscious choice to do the boring version of the story?"

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.

Re: [FRC Blog] Einstein Report Released
 

The idea that there's a person involved has captivated people's attention from the simple points that will still be unresolved regardless of any apology.

The best of the best had power issues that no amount of reading would find.

We have been distracted by the communications about this person from the larger security issue of deauth and other attacks which remain possible.

We've ignored the much more annoying aspect that some of these other attack vectors won't actually stop the robot and leave it stopped they can be used to manipulate the field play (stop a robot at just the right moment just long enough to effect match outcome and in the process make finding the source much more difficult).

It's additionally not clear to me that certain documentation regarding programming is sufficiently direct enough to communicate to teams what things they can do in a particular language that will cause issues but other than that seem perfectly reasonable.

There's so much more to this than just what this person did or other people probably stumbled on earlier in the season.

Re: [FRC Blog] Einstein Report Released
 

I too waited a couple of days before I post. I read the letter, the report and every post on CD regarding this. I agree with Greg and others that the team and the individual should come out in the open and explain what happened. Let's not worry about other people's reaction when they find out. If they want to be angry, let them be angry. If they want to hold a grudge against the team forever, let them do that. If they want to forgive and move on, let them do that. We have no control on what other people think. It should not be a reason whether the individual and the team should come out in the open or not.

I am not making wild speculation here. Based on what I have read and putting my logic hat on, the individual seems to be one of the mentors of one of the Einstein teams. Based on the report and posts on CD, three of the four alliances can be cleared for various reasons. That leaves one alliance of 3 teams. Considering what happened in the semifinal round, there is motive why this individual may want to talk to the FTAs. This part is speculation so I am not going to name the team. I may be completely wrong here.

I think the team will come forward and do the right thing. I also don't think they should accept the free registration for next year. Let's give them some time. The report just came out over the weekend. They probably need to have a team meeting to discuss this and let everyone on the team know and draft a proper statement.

Re: [FRC Blog] Einstein Report Released
 

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

Re: [FRC Blog] Einstein Report Released
 

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

Re: [FRC Blog] Einstein Report Released
 

As I've pointed out before in this topic. It's possible that the power supply that feeds the AP comes up in such a way that causes the router to boot in a state that is not useful.

It's the sort of thing that would depend on what the status of the charge is in the capacitors in the system when you power up. Also it would depend on having a AP a little more sensitive than the median.

I tested a few robots at off season events with tiny oscilloscopes attached to them on the field. Sometimes during a power up it takes a little longer to get to regulation voltage than at other times (we're talking milliseconds max here not seconds).

So it's hard to say that every time you had to reboot an AP it was locked up by a bad processor reset or the exploit. Unfortunately as others have pointed out there are good indicators of the exploit behavior but it wasn't tracked throughout the season.

Hard to say how much was interloper, how much was component malfunction and how much build related failure.

This has come up over and over.

If one assumes that more than one person was aware of the exploit.

There is no assurance that the other people that know are on the team associated with the individual.

In fact the deauth attack (there are 2 issues in the report) could easily have been exploited by anyone anywhere. All you'd have to do is Google it.

So no it's not logical to assume the team had to have known or was the only possible exploiter.

More importantly the individual didn't need to do anything really all that unusual besides be too aggressive attempting to connect to the network.

That's too easy for anyone, even a spectator, to do in mere curiosity.

Once we assume that more than just this individual might be involved who is to say that we can trust that someone somewhere won't do something unfortunate to the individual as it's really the same problem:

1. Everyone who keeps trying to find out is doing something FIRST may not be comfortable with.
2. When someone does find out we don't know how they'll behave.
3. Once the cat is out of the bag we won't know the extent or duration of the consequences metered out to this person.

By actively seeking out this person we are very much doing what we all indicated is bad behavior on their part.

Worse as a community (just like this person's team) we'd take the hit for anyone that went overboard.

We're setting ourselves up for a vicious cycle.

Re: [FRC Blog] Einstein Report Released
 

For clarity, there were multiple root causes identified - some hardware, some software, some robot systems, some unknown, and intentional interference.

Re: [FRC Blog] Einstein Report Released
 

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.

Re: [FRC Blog] Einstein Report Released
 

I think what Nick might of meant, and what I personally am thinking about isn't when FIRST discovered the bug, but when the individual discovered that he/she was able to interfere with matches.

Re: [FRC Blog] Einstein Report Released
 

Al,

Nick was referring to the intentional interferer. How did THEY find the bug, and test it, before using it on Einstein.

Lots of people (including the report) don't buy the person's story that they "accidentally" found out by killing 2056 in SF2-1.

@Ed Law: I hadn't thought of the REASON they came to the FTA's, but now that you say that, I think you're right. They probably suspected that someone else knew, and was targeting THEIR alliance. Otherwise, why would they admit to the crime they'd (to that point, anyway) gotten away with?

Re: [FRC Blog] Einstein Report Released
 

Al,

Sorry, I should have been a little more clear about what I meant. I understand this was an issue with the D-Link robot AP firmware. What I meant is when did the individual themselves find this issue, and when did he prove to himself the existence of this issue with the firmware.

I in no way meant for that to be intended as "Why didn't FIRST know about this earlier?" or something of that nature.

I also want to comment on the "witch hunt" that CD is seemingly brewing. While I would love to hear from this person about the hole itself and how they discovered it, I think they have already paid the price for it. I think for their personal identity to be revealed would only lead to more despair for that person. However, I would like to see a statement from their team over the subject. I think we'll see one soon.

-Nick

Re: [FRC Blog] Einstein Report Released
 

The problem is that other people might have also discovered it through independent means (as Al has acknowledged and I have mentioned word got around prior to Einstein that something might be a problem with the A version of the 1522 so they would have a place to start looking).

More importantly even if you find out this person is the tip of the iceberg, who is to say how straight an answer we'll get about the rest of the story.

Plus if we assume that someone was using it to cheat we'd have to assume they'd not be so silly to get caught otherwise it's of no value to cheat. This person knew that other people were aware of what they might be capable of. Logically all suspicion would flow to this person. It's not the best way to cheat and not get caught.

As others have written, maybe the individual feared the use of this attack against their interests (it doesn't matter either way with regards to their involvement).

With all the other issues I don't think we'll ever know the full extent of how much the 2 wireless issues could have contributed.

Re: [FRC Blog] Einstein Report Released
 

This has been my thinking as well. I don't think I could have worded it any better. My question is, was the Einstein incident the first and only time this person had purposely sabotaged a team?

Re: [FRC Blog] Einstein Report Released
 

Brian,

I did not acknowledge that there is an issue with the A ver of the 1522. The problem is an interaction of the firmware loaded on the Cisco router to fix another problem noted earlier in the season when used with A version. It was discovered in testing after St. Louis that the Cisco firmware and the Dlink AP were affected when used together. Please read page 7 and 8 of the report. Please note that the report also states that this updated firmware was only installed at week four events. That being said, a user testing with wifi devices at their home field could not have discovered this vulnerability. In addition, it could only be discovered at an event using a 5GHz enabled wifi device of which, until recently, there were limited numbers of such devices available.

Re: [FRC Blog] Einstein Report Released
 

I understand completely I read the report. The issue remains that I distinctly remember people telling me they had some unspecified issues with the 1522 version A weeks before Einstein. In fact I have 25's A version 1522 which they thought was intermittent and it's not and they replaced that before they arrived at the championships. That could mean that someone was messing with them or not (all depends on the other factors).

So this means that people had a place to start looking weeks before Einstein. Whether they could find the specific interaction as you said would require a Cisco unit with that firmware (and very few people knew that information). I suppose one could take from that they discovered this by messing with other fields (who knows when or how).

It seems from what I've seen that this person was sure they had something (which is sort of damning).

I was never able to find anything extremely unusual about this 1522 I got from them. So it fits that you'd need this and some external set of factors that do not exist in my environment but do exist on the actual field. Course I can only rely on the information I was told that nothing besides this AP was changed to remedy the issue (otherwise one could argue that something was an issue in their robot as additional stimulus and had changed by the time you reviewed it for this report).

The only way I can think of that someone could have stumbled on this without effecting a field would be to have exactly what's on the field. This seems unlikely given the way Cisco handles firmware and we know the version of firmware matters. That or perhaps they could have compared the 2 versions of the D-Link 1522 AP noticed a change in the behavior which points to the soft spot.

Obviously attacking the field till you find it would be easier and in this case require less technical skill and resources. Course that would almost certainly mean that whoever insisted on bringing this forward had already done something they shouldn't or seen someone else do it.

Re: [FRC Blog] Einstein Report Released
 

In the last paragraph of page 9 of the report, it states:

Then on page 22 of the report:

Could someone with direct knowledge please confirm/clarify: Is the "Field Hardware Logging" mentioned on page 22 specifically referring to retention of the field access point logs mentioned on page 9?

Thank you.

Re: [FRC Blog] Einstein Report Released
 

Brian,
The reports of some people involving ver. A were simply anecdotal, unconfirmed reports that seemed to point in one direction when other things were ignored. If anyone can take anything from the report I hope it is that there are many things that can manifest the same way as an attack that are in fact not related. Go to the report and search for "buffer" to see one of these problems described.

Re: [FRC Blog] Einstein Report Released
 

I grant you this was hardly the only issue. However, those initial reports were enough for me to have both versions of the D-Link AP and compare them. I am persistent and that's a mere $200 to get 2 units to test (nothing I found pointed to the issue). Besides off season events do not get a spare parts kit. So the worst case was I had spare radios to offer in case something went down at the off season events. Besides I was testing my tiny oscilloscopes at off season events and they were on the power into the radio so if something had happened (and it didn't) I could offer a replacement as compensation. When I finished testing my oscilloscopes I merely gave some new 1522 AP away in trade for samples of misbehaving APs.

Obviously I didn't spend all that time and money to build those little oscilloscopes because I thought FIRST merely had AP issues (though I admit that while I knew and still know more ways someone could interfere with the wireless I never thought anyone would be that devious or in this case so easily caught).

Still it leaves it out in the open that at some point, perhaps multiple times, someone tested that exploit before they tried to demonstrate it. Additionally, I do agree with what you seem to conclude that this person didn't intend to rig the rankings. Surely this particular tampering is not the only issue and focusing on it too much distracts from the bigger issues we all must face from it. I'm not sure we'll ever know the full extent of what happened without adequate logs.

Re: [FRC Blog] Einstein Report Released
 

I also agree with what Greg and many others have said, that the individual should come out and admit their mistakes. I have full faith that the FIRST community at large will accept this individual's apology and move on, having learned from this ordeal.

But what I would not like to do is add on to this pressure that's amounting on this individual to speak out. Whether I think he/she should or not is a different matter; the decision is not mine to make. If the individual chooses to remain hidden, he/she may have to deal with the consequences of the discontent team members, and that's up to the individual.

That said,
I think this is very important as well, to keep things in perspective. FIRST is a community unlike the public at large, with the distinction lying on its gracious professionalism. I have no doubt that the community will continue to uphold and cherish this unique characteristic through difficult events such as this.

Re: [FRC Blog] Einstein Report Released
 

Concerning the comment about the forest and the trees, I am not sure what Lil' Lavery is getting at.

Is it that the system worked (i.e. even with all the problems discovered, we had a season with a record number of competitions and and competitors)?

Is it that the system broke but is going to be fixed (i.e. There were problems but FIRST got to the bottom them)?

Is it that the system is broken (i.e. even among robots that make it to Einstein, a substantial proportion have major electrical/programming problems)?

Is it that the system is really broken (i.e. that the control system is a brittle mismash of marginally compatible subsystems that on its best day is can be in inadvertently brought to its knees by well intentioned programmers yet alone nefarious no-goodnicks)?

I can see a lot of forests and a lot of trees.

Joe J.

Is all of the above okay? Cos the way I see it, there are four different systems. In terms of the goal of FRC, we're still doing well, if not better. That's what I call the Chairman's system. Also, the lengths FIRST went to after championships are quite impressive, in what I guess is the organization system. Then, we have the system comprised of the elements of game play, both on the field and in the pits, and all's that proves is how human we are. The Astros sucked once they got to the World Series in 2004, but nobody says that the system is broken (well, at least not for teams sucking at that high a level). Finally, there's the robot connection system, which is the only broken system out of the group. And that's the forest Lil Lav is talking about, with the tree being our unknown attacker. It's not the guy's intentions; it's his actions and how repeatable they might be, how much further a future attacker can go if we don't plan for these things in the future. And it's a tricky line to walk. Do we outright ban cellphones? If you do, all the teams who use social media from the field side or can't afford cameras beyond their phones are being denied a large portion of their team identity. But clearly letting everyone have access is also an issue. So do we fix it on FIRST's side? Make the routers inaccessible? Because that's got a whole bunch of other issues associated with it. So let's talk about those issues, and find a solution to the problem of a guy with a smartphone knocking out robot connections, instead of finding a guy we can yell at for weeks on end.

Re: [FRC Blog] Einstein Report Released
 

I think I get it... let's break this down shall we:

"
I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.
"

"everyone wants to talk about..." This is not true but can be interpreted as an exaggerated expression to bring their point across... Ok that's fine.


"A lot of people" ... this is not you... and really not some that may appear that way now with their posts. For example Greg's post... (I'm picking on him because of the timing of his post and because I have great respect for him)... the post was indeed fixed on one point (i.e. the trees). I should add I know the robowranglers like us (and many teams) will indeed study every piece of this report and make sure that we address each point properly.

So let's just say some people... when it seems like a lot of people.

The last part... missing the forest for the trees.

The reason why I'm going through the trouble of breaking this down is because I got this feeling as well, but it is just a feeling. Basically there are posts that are applying pressure to the perpetrator and/or team, (i.e. the trees)... the forest is all the other problems not talked about as much... I'd say these are the other technical responses some of which I've been trying to say. I know someone else mentioned the idea of splitting the discussion up as well.

I know when the dust settles I will want to discuss some of the other aspects of the report such as the network capping. Right now... I feel like backing off and letting the predominant discussion carry on.

Re: [FRC Blog] Einstein Report Released
 

So based off your interpretation... We as a general FIRST community need to focus on the forest.... Let the specific individuals affected focus on the trees. These are my feelings at least.

Re: [FRC Blog] Einstein Report Released
 

I can't help but feel like this report is somewhat backfiring on FIRST. While I do agree that it has been a vital process and do applaud them for releasing every detail about it, it appears to have created a rift in the FIRST community; splitting people into people who want to move past this at god speed, and a group who, and rightfully so, still attempting to cope with what they've faced. This second group, as exemplified by posts from some of the most esteemed members of the community, is a group most of us will never understand, and hopefully never will have to understand. A sweep of both Worlds and a CCA is, or at least should, be every teams goal, and a once in a lifetime event. To have been so close, and have everything seem to have been ripped away by a single person would be devastating to me. I would never be able to find it in my heart to forgive that individual.

I guess what I'm trying to say is that these teams will need time to cope with what happened, and that the rest of the FIRST community should give them ample time to heal at their own pace. When they're ready to forgive and maybe even forget, that is the time we can all look back at this.

Re: [FRC Blog] Einstein Report Released
 

I have a lot of goals with FIRST, but sweeping the competition merely in search of being the highest scoring team over and over is rarely one of them. It's welcome when it happens but I can't loose sight that this is about overcoming adversity. Those other awards at the end are not merely second, third, forth, etc.

I'm not sure how discussing moving forward harms the people that experienced this. There's nothing that can be done to undo this situation. Perhaps award them additional championship winners but that's beyond my ability to offer.

We do not have the logs to determine how far back that particular individual's attack vector was actually used (or whether they were the only exploiter of it). So the other people that almost certainly were denied beyond the Einstein teams will never know how or why.

We do not know if or when anyone used a deauth attack vector anywhere because again there are no logs.

We do not know how many times a robot made it to a championship or even highly ranked with a technical problem that might be caught with more frequent examination, better tools, or more time to look. How many of those teams were denied on the premise that the top level teams are more effectively mitigating those issues? Only to discover these issues remain at the top of the ranks. No disrespect can be placed on any team as this has been the nature of the competition for a good long time. Equal playing field and equal expectation of common issues.

I'm not rushing them, and I don't think anyone else is either, to forget their pain that would be totally unacceptable. I'm merely pointing out that their pain can't stop the world and their pain can't be a good reason to ignore the direction FIRST has clearly chosen for handling the identity of this team or it's members.

This is a tragic consequence, but the pain of tragedy is often the crushingly simple fact that you carry on and no matter how many times you talk about it will still be a tragedy.

If anyone can find a more suitable memorial for this I'll be happy to contribute but not at the expense of loosing the value of the report or subjecting future teams to a closely related situation.

Re: [FRC Blog] Einstein Report Released
 

I think it is ok to be angry right now. I think it is ok to be straight livid. I think it is ok to be hurt. I think we can be mad for the Einstein teams and for ourselves. We all have skin in this game. We are coaches, players, supporters, and fans. Imagine the outrage if it was found out the Super Bowl was rigged. There would be congressional investigations. It would blow up the 24 hour news cycle. There may even be riots.

Well, this was our Super Bowl, and someone fixed it! Someone tampered with our biggest stage and there are calls to laugh it off and look at the bright side. It is way too soon to be upset with someone for being angry. I was not on Einstein and I am angry! I am angry for those who were immediately affected by this despicable act and I am angry for my team as well. We are competitors. We want to know that our hard work and effort goes into a competition that is truly played on a level field. This person took that away and I am not ready to get over it. There are still too many unanswered questions that need to be resolved. I want to know the whole story, and right now, only a handful of people do.

Einstein is not only important for those involved, but it is extremely important to all of us as well. I read that it is not about the robots, or even about winning, but I disagree. The robots are the vehicle to inspiration about STEM. Winning increases the impact. It is the motivation to improve ourselves. Professionals need to win contracts so they must find the best/most cost-effective solutions. Our goal is to win a competition. We don’t build a robot to do show and tell (the time for that is after the competition season is over) we build one to win Einstein. Competition drives us to improve. We can’t all win Einstein, so when we don’t, we look toward the teams that do make it to serve as our inspiration and motivation for improvement next season. We learn things from the teams involved and try them in the offseason. We use their excellence to make ourselves better. Those teams were cheated from an honest competition, and so were we.

I can see trouble in this thread for us. We are fighting amongst ourselves about what teams should be satisfied with and when they should be ready to move on. Arguments are getting personal and off the subject at hand. Are we really angry at each other, or are we angry at the person who lit this candle. I think this community might be better served by expressing our outrage at the person who did this..by airing it out. I see in this thread and the Sabotage thread that was just closed that we are starting to turn it against ourselves, and I believe it is all misplaced anger. Who are we really angry at?

I am a pilot, and I know from experience that a great flight can be ruined in the eyes of passengers with a bad landing. Bounce one on hard and passengers talk about the whole experience as if it was horrible. Unfortunately, for FRC, Einstein was 2012’s landing. Only time will let the season be put into perspective. Please stop telling people to look at the positives. They will later. If you are ready, good, we need people looking forward, but don’t rush the others. These arguments will only further hurt us, which is exactly what we don’t need right now.

Re: [FRC Blog] Einstein Report Released
 

See other thread: Einstein report powerpoint

Re: [FRC Blog] Einstein Report Released
 

I am sure that most of us are not angry at anyone. I surely am not angry. However, I am concerned that when winning becomes the only acceptable end that the value of the journey is lost. The simple reality is most of us will not win the championship in any given year.

Yes we do aspire to win. We do hope and we do the best we can to win.

Then again many of us will stop like these Einstein teams did and help each other out when it's not in our interest to make sure we win.

Unfortunately, there's clearly a much larger element of chance at work here than we seem to be able to accept as a community. I continue to see people speak of the difficulty of the best of the best teams to reach the highest echelon of competition.

I have been publicly quite pointed about the electrical issues that could impact robots since long before Einstein and long before this report. This report makes it clear that the best of our best still have problems that we've often assumed should not exist at that level of competition.

The combination of problems like the electrical issues from this report and the random uncertainty of the game designs themselves clearly makes it unpredictable that the qualities we think frame the best of the best are any assurance that they'll succeed. Hence the other awards.

The actions of this interloper aside. The uncertainty added to this environment beyond the game design is a fundamental problem that makes this all the more devastating for those most directly impacted. Surely it's an issue that adds more salt to these teams' wounds than is necessary. Even if they do everything just the same as they did this year there's no way they can be sure they'll place at that level again. FIRST's offer to promote them up to at least the venue automatically may only be slight help to them.

To use your example, we hope in the STEM fields when we compete on cost, quality and price we compete on a fair playing field. In the real world we often also compete against politics and tactics that exceed honest business.

I see a lot of frustration in this community which holds STEM values so dear to themselves that we have these uncertainties and lack of logs to find the proofs we so value. I do not think it'll turn to anger or hatred at random. Still I wouldn't fuel that fire with jokes because right now I'm sure some people still need the uneasy peace.

Clearly work needs to be done to limit the random impacts on the game play to those aspects which give opportunity for benefit to those that exhibit the traits we as a competition hold as the best example.

Re: [FRC Blog] Einstein Report Released
 

It is rather sad, but it is my belief that this person will not reveal themself.

Anyone who is willing to interfere with robot communications to try and give their team the advantage is likely a coward. Cowards dont do whats right, they do whats easy.
Its fair to be upset with them...

FIRST has done an excellent job documenting their investigation and for that I thank them.

Re: [FRC Blog] Einstein Report Released
 

Unfortunately, we've seen some of the most disrespectful posts ever in these forums directed at the victims of this incident. Although I can't claim to understand what the members of these teams are going through, I know for a fact that I would be distraught and disheartened for a while following the results of the Investigation if it were me. The last thing these teams need right now is for members of this community to be insensitive to their situation. They are some of the greatest teams comprised of some of the best people that FIRST has to offer.

I know if my hard work was ruined by someone else's wrong doing, I might question the time, effort, and commitment that was invested in it. Even though it wasn't my work that was destroyed, I am still upset, angry, and ashamed that someone in our community would do such a thing. Matters are made worse by the disrespect shown to these teams by their own community. The last thing we need is for some of the best people in our community to question the time, effort, and commitment they devote to FIRST. I fear that if disrespect continues, it could cause even more permanent damage.

Let us unite with support for those effected by the events on Einstein. Please be understanding and allow them ample time to recover from the tragedy, however long that may be.

As for the perpetrator, rather than vilify him/her, an attempt to understand what led him/her to do this would be more appropriate. As Dean Kamen says, "Society gets what it celebrates." What we got on Einstein was an unfair attack. While I know no one celebrates unfair attacks, there had to be something mistakenly celebrated to lead to this result. Speculatively, I would have to say competing at all costs was celebrated where it would be more appropriate to celebrate competing with gracious professionalism. We can't know for sure, so understanding the thoughts behind the actions will serve as a valuable resource to prevent these attempts in the future.

Thank you FIRST, the Einstein competitors, and our volunteers for your thorough investigation of the incident. It makes me proud to be a part of an organization that is willing to go to such lengths for the benefit of our community.

Re: [FRC Blog] Einstein Report Released
 

Quick not I'm sure someone has already brought this up but as a gamer I have made my skin thicker against hackers so that when I hear foul play was suspected I can positively say "then up your game". Is the FMS the best way to run robots I cant say it is because if someone hacks that box they can do what ever they want, they could change the score by adding a few penalties in or seize communication on a bot or even a whole alliance So what is the best system? I don't know. My guess would be something where we didn't have to rely on every single aspect of the game in one software. But that's where you get the "Then up your game" from.

Re: [FRC Blog] Einstein Report Released
 

Reading this made me dizzy. Im sorry but id understand you better if you were to restructure that thought into something more coherent.
You cant relate this to video games, they are totally different in scale and impact.

Re: [FRC Blog] Einstein Report Released
 

I think he's saying that the FMS is one thing controlling everything (scoring, connection, etc...) and that it might be better to have everything be a separate system...but wouldn't all the systems integrated together just be the FMS!??!?! :confused:

Anyway, I think with what he's suggesting we wouldn't be able to have all the stuff we have now with the FMS twitter feed and whatnot.

Re: [FRC Blog] Einstein Report Released
 

This is slightly off-topic but still generally related...

I'm looking for an open channel to someone at FIRST who would be sympathetic to the suggestion that the Twitter data be logged locally to non-volatile storage so it could be made available after-the-fact in those cases where the feed is blocked at the event.

There are amateur statisticians and historians in the community who would love to have complete data for analysis.

Re: [FRC Blog] Einstein Report Released
 

I don't believe the report says anything about the FMS being hacked, scores being changed, or robots being seized. There is no evidence those actions occurred on Einstein.

The report discusses how the FIRST staff performed some typical DoS attacks on the bridge and router to learn what the symptoms would look like. The report discusses that a bug was discovered in the field wifi components that allowed for a disruption of service. The FIRST staff then explored the various symptoms and the requirements for the bug to manifest. The bug allowed for service disruptions, but no foreign device joined any field access point.

Also, the exploit required no hacking skills. Hackers everywhere are cringing when this is referred to as a hack. The term hack never appears in the report. Sorry to be such a stickler for terminology, but inaccurate descriptions of what took place do not help matters. If there are parts of the report which need clarification, please ask rather than jump to conclusions.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

The FMS was hacked in the way that 14 year old's Facebooks are "hacked" by their best friend who posted, "I smell" while they are both in the same room.

(I've been waiting for someone to post something like that so I can make the joke :o )

Re: [FRC Blog] Einstein Report Released
 

There are 2 vectors in that report.

The confirmed vector was the one that needed very little beyond a phone. They found someone that admitted to that on Einstein.

The other vector (which does work but we have no evidence either way it was used) was deauth and generally that one is described on hack a day in October 2011.

As a person that works in computer security I know most big bad 'hackers' people find are just exploiting the much more time consuming efforts of others.

In this person's case it is more social engineering. They must have tested this before they reported it. The manipulation is in reporting it in such a way we will not be able to find out how and when that was done before.

Course they may not have realized that there were insufficient logs stored on the field servers so that was a gamble.

Still there is no evidence presented to support the idea that this person intended to influence the Einstein matches in a particular direction (who got hit was just a function of proving it worked at all).

The trick with the phone wouldn't have worked on at least 2 of the robots because those 2 had the B version of the D-Link AP on them at the time. Without logs we have no way of knowing whether the person with the phone knew that those 2 teams had that B version AP and ignored them in their effort. So there's no reason to suspect that this person knew anything more than this trick they pulled worked before somewhere, somehow.

Keeping in mind that this trick with the phone also requires the Cisco field AP to have a specific version of firmware the only practical place to test that without heavy reverse engineering would be on a field.

Re: [FRC Blog] Einstein Report Released
 

Does the report say that they reported it? I believe it says they admitted it.

I think the appropriate place for security experts to report vulnerabilities would be directly to FIRST staff. If you have a knack for hacking, social or otherwise, do the right thing -- wear the white hat.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

This. This. A million times, this.

If you're clever enough to figure out things like this, you should be smart enough to realize you have a duty to keep this knowledge out of the wrong hands and disclose it properly.

Re: [FRC Blog] Einstein Report Released
 

Ah HA! Those two robots must have been on the Blue alliance!
;)

Re: [FRC Blog] Einstein Report Released
 

In the report it says this:

Page 10:

"While the Einstein matches were in progress, an individual was observed near the field using a cell phone in an apparent attempt to access the field WiFi network. This individual had attempted to engage field personnel in discussions while the field personnel were troubleshooting other issues. This individual was asked to put away the cell phone, and complied. Later, the individual was observed using the cell phone again, and at that point, before the last two Einstein matches were played, was asked to leave the field area, and did so."

"After Championship, this individual came forward wishing to share knowledge regarding the failed client authentication issue. The individual claimed to have attempted to connect to the network associated with Team 2056 during Semi-Final 2-1 and observed that this attempt corresponded with the robot losing communication."

Posts were made with slightly more information. That was pages ago and I'll leave that to someone else to cull.

I would have to suspect that the attempt to engage field personnel was their initial attempt to report the issue.

Please be aware that there are currently several vectors I could shut down: all the robots, some of the robots, or halt any of your robots at will.

Remember I do computer security and it's part of my job and no one asks me my opinion about this at FIRST.

Additionally I know other people have reported vectors some of which are not listed and have yet to get a clear line to FIRST either.

Your assumption presumes that you can get the ear of the person you need to talk to and that in reporting it someone decides to mitigate it.

So for example perhaps someone tried to be the 'white hat' when they had a person face to face at Einstein and didn't like what they got for it (I presume only...it could also be that they were just trying to cozy up to field personnel to get better signal strength).

Appendix A of the report is a little tricky to read because of how it's arranged in the left column.

Teams 16, 25, and 207 had the B version and would not have been effected by the stunt with the phone trying to connect.

Re: [FRC Blog] Einstein Report Released
 

I'm sorry for paraphrasing, but at least one of the field personnel was an NI employee acting as CSA. No, it wasn't me, but I've heard this story from several points of view. I have to believe that there are better ways to gain signal strength. If you have access to Bill Miller, Frank Merrick, the FTAs, and virtually all of the FIRST staff wearing red ball caps, I think you can increase signal strength in many ways.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

It appears Marty McFly's Time Machine/Delorean would be useful now. It would be truly wonderful to go back and change these events and fix them before they had a chance to happen. :) Instead we are left with trying to prevent them in the future. I will leave this to the programmers and engineers.
I have followed this thread all week. For good reason, there is much anger and those effected have every right to feel this way for many reasons. As more than 1 poster has stated, this was a chance to have a blue flag for once. Just an idea to consider, may be a really bad one, but I will put it out there anyway just from a Mom's perspective. Could a flag/banner be created for this year's finalists that is extra special/limited edition/special color variety? Given to all that were in the finals since each might have had a chance at the title. Sort of a badge/banner of honor, for surviving and enduring the 2012 catastrophe? As the years go by, it would be special as no other team will have the chance to have such a banner, and the teams that go on can take pride if the fact they persevered and moved forward. I am in no way suggesting this makes up for what happened or replaces a title championship. I am suggesting this pays tribute to the teams who have endured much and are to be respected.
It is a small gesture but sometimes small gestures are very meaningful. I understand there would be an extra cost issue here which would need to be addressed to provide these teams with these banners, but I bet many of us who feel for these teams would gladly help. Again, just a Mom's suggestion and if some of you find it insulting or inadequate please forgive me, I promise I mean well. :) The only way to ever make a positive out of a very bad situation is to learn from it or make it better for those coming along next. Wish I had the perfect words or solution, but nobody does. FIRST will continue and let's try to make it better for future FIRSTers.

Re: [FRC Blog] Einstein Report Released
 

If in this case you equate 'signal strength' with finding someone who will listen then the question is how in someone's face can you get during a crisis (there were other issues at work). I don't know the particulars of the venue that day. I was at work in NYC watching over the Internet.

In this case I meant literal signal strength. Their ability to connect to the network over the any possible interference from other people would be improved by proximity to the field. Something you might be able to get by giving yourself a way to get close to it (say by talking to field personnel).

No matter what if they attempted to report it at all it would mean they tried it before that or saw someone else do it.

More importantly some of these attacks mirror the field troubles we noticed at Monty Madness off season the year before.
That was another example where I personally checked the D-Link AP and the robot and there were no problems.
Go out on the field and poof...

I did mention that in the topic about alternate control systems where you explained about the new logging features before the start of all the 2012 events.

I suspect this has been brewing for a while.

Re: [FRC Blog] Einstein Report Released
 

Kudos to FIRST for the attention to detail, thoughtfulness, and completion of this report. The effort of FIRST, the volunteers and suppliers, and the Einstein 12 is much appreciated.

I really can't understand quite how the teams on Einstein feel about all of this, but I can understand the frustration, bewilderment, and anger. While the investigation is going to make things better, this situation is still tough to handle and is very frustrating. The sabotage was not just to certain teams, but to the entire FRC program and community.

Everyone builds their robots uniquely, runs their teams differently, and prefers all kinds of flavors of ice cream. Now, we are seeing that folks vent and react differently to a frustrating situation. ... and that's ok.

Andy B.

Re: [FRC Blog] Einstein Report Released
 

Disclaimer before clicking the below link: The youtube video contains some foul language. I copied at a time stamp past that portion of the TED talk.

Been following this thread, with a TED talk by Anthony Robbins ringing in my head. Everyone reacts to all situations differently. Some people will find silver linings to anything. Some people will point fingers. Some want heads to roll, others want to give a hug.

We are role models. Not just our mentors, our entire community to the rest of the world. However you react, make sure whoever is watching you is inspired by it in a positive way.

Re: [FRC Blog] Einstein Report Released
 

If 118's loss of comms was because of the programming loop involving the gyro, I have one question.

Did they experience these issues before Einstein? did they just load new code before Einstein? Or is it a possibility that the issue with the wire crimp just decided to pop up conveniently as all the robots were dropping comms like flies?

I could not find an explanation for this in the report...am I missing something?

Re: [FRC Blog] Einstein Report Released
 

I'm obviously not affiliated with them, but I believe at regionals(CT and Alamo specifically) they suffered from comm issues that strongly hurt them. No idea if it was the same problem though.

Re: [FRC Blog] Einstein Report Released
 

The CT eliminations (in week 5) saw multiple robot loss of control events from several good teams, some of which were clearly robot system issues (IIRC faulty battery connector, USB hub disconnection) but also others which had no clear cause identified at the time.

As Steven suggested it's possible that 118 suffered from the same subtle and unfortunate system interaction bug that was discovered before the Einstein investigation weekend.

But one of the effects of the Einstein revelations on us all as a community is that now there has to be some suspicion that a similar act of interference occurred at Hartford.
Innocence lost...

Re: [FRC Blog] Einstein Report Released
 

I did not attend CT, but worked with 118 the following week in Houston, trying to identify the issue. I looked at the logs from CT and they looked very similar to Einstein. It is very likely that the sensor connection led to the failure in Houston and the ones in CT as well. They made code changes in Houston, I don't believe they did so before Einstein, and I don't believe they introduced the problem between divisions and Einstein. The code issue was present, lurking for a long time.

If the sensor connection had never failed, the loop in the init code would do what it normally did and the robot would have operated wonderfully.

If the sensor connection had failed permanently, they would have hooked up a complete debugger in the pits, located the loop and the sensor, fixed them both lickity-split, and operated wonderfully afterwards.

But the sensor connection apparently failed just a few times during the season. Perhaps it was brought on by the cart or the loading or reset procedure, or vibration, but since it didn't stay in a failure state, the chance to debug was fleeting. Additionally, the sensor wasn't used and the init code wasn't executed unless the testing included the auto-tele transition. I believe this was another factor that influenced how team 118 interpreted the cause. Bugs that are difficult to reproduce are incredibly frustrating in all disciplines.

This is one of the reason why it is important to think a lot about debugging, and to consider building harnesses and platforms and procedures that enable you to test your devices well. Most things in the world don't work the first time, they interact in ways you didn't predict, and they may change over time or under different conditions. Managing that chaos is a part of what engineers do.

118 is a great team that builds great robots, but this time both Murphy and Achilles had their influence. I look forward to working with them in the future.

Greg McKaskle

Re: [FRC Blog] Einstein Report Released
 

Yes I agree... we have a completely different simulation to run our code. To add to this... I never... never write an infinite loop (and hope for some breaking condition). There should never be a case for that, but I see programmers do it (even some of the best).

Just to be clear... I define an infinite loop as this:
while (true)
{
if (I hope this works) break;
};

You can always do this instead

while (timeout++ < threshold)
{
if (I hope this works) break;
}
if (timeout >= threshold)
{
assert(false);
error recovery here;
}

Also any thing inside the autonomous loop (in this case) could check for
(IsAutonomous() && !IsDisabled())
to determine if autonomous is still happening... in my previous entry it is absolutely critical to exit autonomous loop ASAP... otherwise you'll lose telop connection for a good chunk of the match. (This is worth repeating).

If anyone knows of that issue being fixed please let me know... According to this report it sounds like it still is the same as it was last year.

Re: [FRC Blog] Einstein Report Released
 

After reading through the Einstein report and this thread several times; I had to digest all the information before making a post.

It is very unfortunate what happened on Einstein. I was home watching as my wife was screaming at me because we were supposed to get to dinner. I was very excited because most of the team that were on Einstein this year are my “friends” and they have done enough for team 108 for the last several years. I was ready to watch them tear it up on the field instead I sat there staring at dead robots.

I have nothing more to say than I am disgusted by the individual’s action. The teams affected have every right to be upset and frustrated. Heck, I still hold grudges against a mentor that came into our pit couple years ago and thought it would be gracious to disconnect all the pwm cables from the jaguars. It is unfortunate that situations like this happen in our community. I bet the Einstein teams attending IRI this year will bring their A game and I can’t wait to see them all compete in person.

Re: [FRC Blog] Einstein Report Released
 

We did not change any code before Einstein. As Greg states, this was likely an issue that was with us all season. While we know the gyro reporting bad data was the start of our error chain, we’re not 100% on why we got that bad data. The way we could replicate this failure post Einstein was by unplugging the gyro (no data = bad data). The report states a faulty crimp is the most likely cause. This may in fact be true, however, it’s a little hard to accept due to seeing the failure so infrequently and having no issues in between. For instance, we saw one of these failures on practice day of Houston. Without touching the robot, we were able to run another practice match immediately after with the exact same robots on the field. This match ran without issues.

After this practice match in Houston, we replaced the cRIO. We didn’t see another occurrence of the problem until the first match on Einstein. Obviously we thought the cRIO fixed the problem and there was no need for further troubleshooting.

Looking back we can only assume that the problem outlined in the report was the cause of the one failure we had practice day in Houston and the two semi-final matches in Connecticut. The second leg of the error chain was part of the code. After Connecticut, our programmers literally spent hours looking for a spin loop without an exit. It simply wasn’t obvious and they didn’t find it.

As Greg alluded to, one big lesson learned for us is how to better perform our full robot checkouts, which we do before and after every match. We did NOT include a run through autonomous mode during these checkouts, which meant we would never have seen this failure. Had we done this, we may have seen the failure more often and been able to better diagnose it.

We learned some valuable lessons from this, and are very thankful to Greg and the FIRST team up in NH for their work with us and all the Einstein teams.

We know how horrible it feels to have our robot not compete due to reasons that ultimately were our fault. We’re sure it must feel much worse to those teams who were brought down through no fault of their own. We can’t begin to express how saddened we are to know that this happened to our fellow competitors.