[FRC Blog] Einstein Report Released

[plnyyanks]

Originally Posted by CalTran

Does that mean we're having another unofficial ChiefDelphi web hug??

Web-ier and hug-ier than ever before! But wait! There's more! Call within the next 10 minutes and we'll throw in an extra hug FREE! (just pay processing and handling)

[OliviaG]

Originally Posted by plnyyanks

Web-ier and hug-ier than ever before! But wait! There's more! Call within the next 10 minutes and we'll throw in an extra hug FREE! (just pay processing and handling)

Wow a web hug??

[DonRotolo]

Virtual, of course.

I've waited until the report digested properly to comment.

The short version is: Good job everyone, that's a good report. Thank you. 1676 suffered lots of comm problems throughout the season, and I'm 98% convinced that our code is at fault.

Now for the hard part: Finding it...

[Greg Needel]

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

I was standing by the field during Einstein with no team in the competition, only for the opportunity to be there with friends from other teams and try to give them support in the finals. I can honestly say I have never had such a range of emotions brought on by any series of events then these finals. I witnessed the discussions, I vented my frustrations, and I offered my sympathies and congratulations to my friends. One thing remains is that I have never been so angry at a robotics event, to the point that I felt physically ill after the event. ALL THIS WITHOUT MY TEAM COMPETING. My heart will always go out to my friends on the teams that will have to live with the feelings of being taken advantage of, and the thoughts of "what if" for the rest of their lives. Sure, the pain will lessen but you never forget these things.

Afterwards many people, including myself, second guessed how FIRST handled the situation and wished that things had been done differently regarding on the spot trouble shooting, but not having been in their shoes and knowing that hindsight is always 20/20, I have nothing to say on this aspect of the situation.

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure. In all honesty, part of me wishes that it had just been a random occurrence caused by almost anything else. The fact that someone knowingly sabotaged the matches is UNFATHOMABLE in the community which we have. Not only has this person impacted the lives of the teams he stole from, but has brought into question how trusting we can be with our community. I hope that this is an isolated incident, but sabotage or interference will now be in the minds of every competing team for both future and past events.

The one thing that is left still unresolved in this whole situation is a public statement from the individual and team involved.

The person who did this needs to stand up for his actions, not just to provide a sense of closure to the situation, but also to grow from his mistakes. One of the hardest things in life a person has to do is admit that they were wrong, and while the punishment has already been dealt out, it is a showing of personal maturity about the way the situation should be handled. When you make a mistake you must come clean for both the people impacted by your decisions, and to clear your conscious. In addition to a public apology I believe the right thing to do would be to reveal each instance that this exploit was used in order to clear the air and remove the doubt that has clouded the minds of every competitor at events from week 4 onward.

It is going to be a rough road if (and hopefully when) you do come forward, but you should sleep in the bed you make, and in the end I promise you that you will be better person for it.

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.



Greg

[AdamHeard]

Originally Posted by Greg Needel

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability from your actions.



Greg

Greg, I really respect this post and your opinion.

[Akash Rastogi]

Originally Posted by Greg Needel

it is neither gracious or professional to not have personal accountability for your actions.



Greg

+1

This last part of your post really stood out to me. Accountability is also a way for you to earn at least a little respect back from those you have wronged; some sort of decency is kept. The speculation and rumors that spread amongst the community may also sometimes breed animosity for the entire team, regardless of who took part in the wrong-doing. The longer this team/person holds out on some form of apology, the less and less respect, forgiveness, and sense of decency they should expect from the FIRST community and especially the teams they affected.

On another note- just read tonight that a team who will be attending IRI was able to fix bugs in their robot thanks to the Einstein report and I hope this is the case for many others, so +1 to all those who took part in the testing.

[AlexD744]

Originally Posted by Greg Needel

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

I was standing by the field during Einstein with no team in the competition, only for the opportunity to be there with friends from other teams and try to give them support in the finals. I can honestly say I have never had such a range of emotions brought on by any series of events then these finals. I witnessed the discussions, I vented my frustrations, and I offered my sympathies and congratulations to my friends. One thing remains is that I have never been so angry at a robotics event, to the point that I felt physically ill after the event. ALL THIS WITHOUT MY TEAM COMPETING. My heart will always go out to my friends on the teams that will have to live with the feelings of being taken advantage of, and the thoughts of "what if" for the rest of their lives. Sure, the pain will lessen but you never forget these things.

Afterwards many people, including myself, second guessed how FIRST handled the situation and wished that things had been done differently regarding on the spot trouble shooting, but not having been in their shoes and knowing that hindsight is always 20/20, I have nothing to say on this aspect of the situation.

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure. In all honesty, part of me wishes that it had just been a random occurrence caused by almost anything else. The fact that someone knowingly sabotaged the matches is UNFATHOMABLE in the community which we have. Not only has this person impacted the lives of the teams he stole from, but has brought into question how trusting we can be with our community. I hope that this is an isolated incident, but sabotage or interference will now be in the minds of every competing team for both future and past events.

The one thing that is left still unresolved in this whole situation is a public statement from the individual and team involved.

The person who did this needs to stand up for his actions, not just to provide a sense of closure to the situation, but also to grow from his mistakes. One of the hardest things in life a person has to do is admit that they were wrong, and while the punishment has already been dealt out, it is a showing of personal maturity about the way the situation should be handled. When you make a mistake you must come clean for both the people impacted by your decisions, and to clear your conscious. In addition to a public apology I believe the right thing to do would be to reveal each instance that this exploit was used in order to clear the air and remove the doubt that has clouded the minds of every competitor at events from week 4 onward.

It is going to be a rough road if (and hopefully when) you do come forward, but you should sleep in the bed you make, and in the end I promise you that you will be better person for it.

According to the report this person has come forward to FIRST, which means that there are people who know their identity, and I doubt something like this will stay a secret forever. I would highly recommend this come out on their own terms then in some other manor, especially with the emotions the way they are.

Another thing that I would like to see is the team step forward and issue a public apology for the behavior of one of it's members. I believe it is not clear if the team knew or if it was one person working alone, which is something I think needs to be resolved. I know that there are deep risks associated with a team and program being tainted by this reputation exist, but similarly to above I have no doubt that the information will eventually come out and it is always better to be proactive then reactive. Additionally there is a great deal of speculation that the person was associated with one of the Einstein teams, and if that is the case, at a minimum, they should absolutely not accept the waved entry fee into next years championship.

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.



Greg

This, literally this.

It is healthy to admit your mistakes. No one is perfect, we have all done things wrong. It takes a big person to understand their mistakes and admit that they were wrong. Too many people in the world believe that they are perfect, or good enough and have it all figured out, and it's just not right or healthy.

Furthermore, in addition to what Greg said, I ask that if and when a public apology comes out, that the first community would be able to do an even harder thing: forgive the individual. Just as it is unhealthy to harbor your wrongdoing in secret, it is just as unhealthy to hold a grudge against those who have wronged you.

This individual is not the only one who needs to be prepared for when (hopefully when) the public apology comes out. Is each individual in the first community, and more specifically the 12 Einstein teams, ready for the apology? Are you ready to realize that what's done is done, to forgive the individual, and work to ensure that this does not happen again? I think each person invested in this community needs to ask themselves this question.

Although it is hard to forgive and it's natural to be angry, there is no benefit in holding the hatred in your heart.

~$0.02
Alex

[Lil' Lavery]

"You understand we're making a conscious choice to do the boring version of the story?"

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.

[techhelpbb]

Originally Posted by Lil' Lavery

"You understand we're making a conscious choice to do the boring version of the story?"

I realize that everyone wants to talk about the controversial and shocking part of the report, but a lot of people are missing the forest for the trees here.

The idea that there's a person involved has captivated people's attention from the simple points that will still be unresolved regardless of any apology.

The best of the best had power issues that no amount of reading would find.

We have been distracted by the communications about this person from the larger security issue of deauth and other attacks which remain possible.

We've ignored the much more annoying aspect that some of these other attack vectors won't actually stop the robot and leave it stopped they can be used to manipulate the field play (stop a robot at just the right moment just long enough to effect match outcome and in the process make finding the source much more difficult).

It's additionally not clear to me that certain documentation regarding programming is sufficiently direct enough to communicate to teams what things they can do in a particular language that will cause issues but other than that seem perfectly reasonable.

There's so much more to this than just what this person did or other people probably stumbled on earlier in the season.

[Ed Law]

Originally Posted by Greg Needel

I have been thinking about how to properly phrase my opinions so that they are interpreted correctly, so here goes.

...

While what I am suggesting may be interpreted as "not GP" I want to remind folks that it is neither gracious or professional to not have personal accountability for your actions.
Greg

I too waited a couple of days before I post. I read the letter, the report and every post on CD regarding this. I agree with Greg and others that the team and the individual should come out in the open and explain what happened. Let's not worry about other people's reaction when they find out. If they want to be angry, let them be angry. If they want to hold a grudge against the team forever, let them do that. If they want to forgive and move on, let them do that. We have no control on what other people think. It should not be a reason whether the individual and the team should come out in the open or not.

I am not making wild speculation here. Based on what I have read and putting my logic hat on, the individual seems to be one of the mentors of one of the Einstein teams. Based on the report and posts on CD, three of the four alliances can be cleared for various reasons. That leaves one alliance of 3 teams. Considering what happened in the semifinal round, there is motive why this individual may want to talk to the FTAs. This part is speculation so I am not going to name the team. I may be completely wrong here.

I think the team will come forward and do the right thing. I also don't think they should accept the free registration for next year. Let's give them some time. The report just came out over the weekend. They probably need to have a team meeting to discuss this and let everyone on the team know and draft a proper statement.

[Nick Lawrence]

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

[Camren]

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

[techhelpbb]

Originally Posted by Camren

Ummm anyone else find that the Einstein FMS used at the prior events Chesapeake Regional, Virginia Regional, Midwest Regional and 10,000 Lakes Regional weren't perfect? Team 3081 had to restart their router going into every elimination match at 10k lakes. Other than that I thought the investigation report was carried through well.

As I've pointed out before in this topic. It's possible that the power supply that feeds the AP comes up in such a way that causes the router to boot in a state that is not useful.

It's the sort of thing that would depend on what the status of the charge is in the capacitors in the system when you power up. Also it would depend on having a AP a little more sensitive than the median.

I tested a few robots at off season events with tiny oscilloscopes attached to them on the field. Sometimes during a power up it takes a little longer to get to regulation voltage than at other times (we're talking milliseconds max here not seconds).

So it's hard to say that every time you had to reboot an AP it was locked up by a bad processor reset or the exploit. Unfortunately as others have pointed out there are good indicators of the exploit behavior but it wasn't tracked throughout the season.

Hard to say how much was interloper, how much was component malfunction and how much build related failure.

Originally Posted by Nick Lawrence

I had to wait a few days to post as some portions of this report have disgusted me beyond belief. I won't comment on that.

I am happy to see that a white paper is going to come out thoroughly describing how the FMS works. I applaud FIRST for this.

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein? (If it happened at champs in this team's division, could that division have been different, if the attack was used there?) I saw some "unexplained" complete control losses at champs this year that could (not saying 100% for sure, not even 1% for sure,) be attributed to a FCA issue.

I really also would love to hear from the team this individual was from. It would be better for them in the long run, rather than people thinking the whole team is made up of "cheating individuals" when I'm pretty sure that isn't the case.

-Nick

This has come up over and over.

If one assumes that more than one person was aware of the exploit.

There is no assurance that the other people that know are on the team associated with the individual.

In fact the deauth attack (there are 2 issues in the report) could easily have been exploited by anyone anywhere. All you'd have to do is Google it.

So no it's not logical to assume the team had to have known or was the only possible exploiter.

More importantly the individual didn't need to do anything really all that unusual besides be too aggressive attempting to connect to the network.

That's too easy for anyone, even a spectator, to do in mere curiosity.

Once we assume that more than just this individual might be involved who is to say that we can trust that someone somewhere won't do something unfortunate to the individual as it's really the same problem:

1. Everyone who keeps trying to find out is doing something FIRST may not be comfortable with.
2. When someone does find out we don't know how they'll behave.
3. Once the cat is out of the bag we won't know the extent or duration of the consequences metered out to this person.

By actively seeking out this person we are very much doing what we all indicated is bad behavior on their part.

Worse as a community (just like this person's team) we'd take the hit for anyone that went overboard.

We're setting ourselves up for a vicious cycle.

[Chris Fultz]

Originally Posted by Greg Needel

I am extremely happy that the investigation was done thoroughly, and like many of you I am extremely disappointed on the root cause of the failure.

Greg

For clarity, there were multiple root causes identified - some hardware, some software, some robot systems, some unknown, and intentional interference.

[Al Skierkiewicz]

Originally Posted by Nick Lawrence

What really burns in my mind right now is when was this bug discovered? More importantly, how was it tested before used on Einstein?

-Nick

Nick,
The bug was not discovered until well after St. Louis. Following an investigation and suspected cause, FIRST engineering staff began a testing sequence prior to the Einstein Weekend. It was during that testing that the interaction of all the components was found. Please remember it is the specific Cisco firmware, with the Dlink AP of a certain type that suffers the vulnerability.