FIRST is really looking into the Einstein problems

[Bruceb]

Here is an e-mail I just got from FIRST
Dear FRC Teams,

Please allow me to update you on our examination of the Championship final rounds.

As part of the examination, FIRST will evaluate the field and all robots that were on Einstein as a complete system—a system as comparable as possible to what existed at Championship. We have received and reassembled the Einstein field equipment at headquarters in Manchester, and we are in the process of retesting its functionality. In addition, we are asking each of the twelve teams that competed on Einstein to (1) allow FIRST to ship their robot to Manchester, (2) provide an adult representative to be present for testing, and (3) provide detailed systems information, including driver station logs and robot code. Each of the teams has acted with the utmost grace and professionalism, and for that we are truly grateful.

We also recognize that many of you have important information to share about your experiences this year. Thank you to those who have passed along such information in response to my April 29th e-mail. To make that process easier, we have set up an e-mail for you to share your experiences and expertise: 2012FRCfeedback@usfirst.org. Please know that while all of your feedback will be reviewed and considered, FIRST will not be able to respond to every e-mail.

We are also investigating the issues on Einstein with independent, outside experts as well as respected leaders in the FIRST community. We remain steadfastly committed to understanding the root causes of the issues and resolving them. I will continue to report our process, progress and eventually the findings of the examination.

Sincerely,

Jon

[R.C.]

Awesome,

Great step forward!

-RC

[Chris is me]

Whoa. This is fantastic. They're even acknowledging, potentially, the issues outside of Einstein field by soliciting input from teams. What a great step forward.

[Jon Stratis]

The lengths their going to get this figured out is certainly impressive. I really feel for the difficulty of their job here... It's quite possible they get the robots on the field and run all day and have no issues. What if the problem was a loose wire that's now tight since the field has been disassembled and reassembled? What if it was having thousands of cell phones in the arena next to the field? What if it was some specific environmental problem in the dome that isn't present now? Hopefully bringing back those same teams helps them reproduce the issue reliably!

[JB987]

Now that the cat is out of the bag...
Headquarters has actually told teams that they can also send one student team member with an adult if that student is a critical part of the programming/control of their robot. We are sending our programmer/driver and our programming mentor to assist with the effort taking place on June 9 and 10. It's a long way from Vegas but it will be worth the trip if the effort provides useful information.

[dag0620]

Bravo FIRST. Bravo.

[JVN]

Kudos to all the Einstein teams for their support of this process! I guess no one should be surprised that the "final 12" would all be pinnacles of professionalism. :)

[IKE]

The lengths their going to get this figured out is certainly impressive. I really feel for the difficulty of their job here... It's quite possible they get the robots on the field and run all day and have no issues. What if the problem was a loose wire that's now tight since the field has been disassembled and reassembled? What if it was having thousands of cell phones in the arena next to the field? What if it was some specific environmental problem in the dome that isn't present now? Hopefully bringing back those same teams helps them reproduce the issue reliably!

Remember, this is an effort (a huge effort) to either find a solution, or remove a lot of "what-ifs".
I am really impressed by the efforts from all.

[Steven Donow]

Awesome job FIRST! I never had a doubt in my mind that FIRST would go to such great measures to solve this problem.

[CalTran]

Is there a "like thread" button somewhere?

[normpear2]

Is it possible if we can get a list of what regionals, if any, the Einstein Field was used at?

[IndySam]

Is it possible if we can get a list of what regionals, if any, the Einstein Field was used at?

I wasn't used at any regional. It was kept in reserve in Memphis for emergency purposes.

[Grim Tuesday]

Is it possible if we can get a list of what regionals, if any, the Einstein Field was used at?

Einstein Field was never used at a regional. It spent the season stored in Tennessee, waiting for the catastrophic failure of another field, to replace it if need be.

[CalTran]

Einstein Field was never used at a regional. It spent the season stored in Tennessee, waiting for the catastrophic failure of another field, to replace it if need be.

Ironic, all things considered.

[dcarr]

Is it possible if we can get a list of what regionals, if any, the Einstein Field was used at?

The Einstein field is kept as a backup and not used until Championship, if I'm not mistaken...

Props to FIRST for taking these steps, there sure was a lot of criticism after the Championship and I'm glad their serious about figuring out the problem. It it means a serious overhaul of FMS and utilizing something better than consumer-grade WiFi radios, I'm all for it-hopefully they're willing to take that step, if necessary.

[normpear2]

Einstein Field was never used at a regional. It spent the season stored in Tennessee, waiting for the catastrophic failure of another field, to replace it if need be.

That actually could explain why this happened.


Lack of being tested in competition conditions. There is a reasons most products are heavily tested before being used.

[Ziv]

That actually could explain why this happened.


Lack of being tested in competition conditions. There is a reasons most products are heavily tested before being used.

This has been brought up. It's certainly a possible factor, but mysterious communications issues occurred at several events—including some of the divisions—throughout the season. It's also unclear what would be different about this field from the others (something that I'm sure will be investigated if necessary in Manchester).

[Chris is me]

I personally don't believe the issue was contained to a single field this year. I obviously have nothing to prove that point that isn't circumstantial, but I think making the assumption that this is an isolated incident that was unique to Einstein is not a safe one to make at all.

[jblay]

Are the teams travel and hotel costs being paid for by FIRST? I sure hope so.

It seems like this email is kind of finally admitting it was a greater problem than just Einstein by asking teams to email in the issues they had outside of Einstein which I think is crucial to not only finding the cause of the issue but also restoring people's faith in the way FIRST deals with these issues.

[sgreco]

I personally don't believe the issue was contained to a single field this year. I obviously have nothing to prove that point that isn't circumstantial, but I think making the assumption that this is an isolated incident that was unique to Einstein is not a safe one to make at all.

I'm pretty sure this is correct. There were plenty of other connection problems at a bunch of events.

That said, FIRST is doing the best they can, and they certainly are not shying away from the problem at all. The most we can hope for is that FIRST finds not just a problem on the Einstein field, but possibly the derivative of all of these connection issues so that it can be corrected for next season.

[normpear2]

It could be the lack of available empty Channels for WiFi to broadcast over. There are, even when teams are ask not to, a large number of open wi-fi networks, that, even if they aren't connecting, could be causing slight interference across the channel their network is running on.

Well, that is what I can think off. The only thing I'm leaning toward is some kind of outside interference. They really should test the field with large numbers of wifi and cell phones in the area and see if that causes any effects or not, I know from the past, that when sitting even at the top of the stands, you can see the routers for the robots on the field, and I'm talking 4/5 bar signals, not 1 or 2.

The dlinks are 5 Ghz Compatible, maybe its time to try it @ 5? There would be less interference due to the lower number of 5 Ghz compatible computers and other devices. We could run 2.4 Ghz @ home, and use 5 GHz at competitions as part of the router setup kiosk, since you are only allowed to connect via bridge or tether at competitions, it would cause no problems other than the practice fields would need Dlinks instead of the old Cisco routers.

I mean, 99% of these comm issues don't happen outside the field.

It could even be the FMS itself...

[Zebra_Fact_Man]

I'm so proud of FIRST that they have the honor to admit when something went wrong and proactively seek out an answer, a cause, and a solution. The field this year was problematic and I hope for the sake of fair competition that they get this thing fixed.

[Meredith Novak]

The Einstein field elements were new and unused, the FMS had been at several prior events.

[topgun]

So, one fully functional FIRST Field and 12 of the best robots in the world with pretty much unlimited time to "test". That would be like the ultimate playground. I volunteer! :D

[JB987]

"Are the teams travel and hotel costs being paid for by FIRST? I sure hope so."...Travel, hotel and meals

"unlimited time"?...nah, just two days, June 9 and 10

[Grim Tuesday]

The Einstein field elements were new and unused, the FMS had been at several prior events.

That's new information, Meredith. Very interesting; do you know which events it was at?

[Ether]

proactively

wrong word choice.

[AmoryG]

I wonder if there's a way to simulate the bad interference the weather outside may have produced?

[jyh947]

I wonder if there's a way to simulate the bad interference the weather outside may have produced?
Hire Thor to mess around for a day or two. :D

With seriousness, I'm super happy that FIRST is taking these steps to re-enact the problems that Einstein had this year. As a spectator, I was very disappointed with all the field issues that plagued the finals matches (I really wanted to see 118's robot shoot!).

I do hope that FIRST will also compensate (partly or fully) the 12 adults for the cost of a trip to Manchester. Taking a trip from Florida or Nevada doesn't sound so cheap, not to mention the money lost from taking time off work...

[JB987]

Hire Thor to mess around for a day or two. :D

With seriousness, I'm super happy that FIRST is taking these steps to re-enact the problems that Einstein had this year. As a spectator, I was very disappointed with all the field issues that plagued the finals matches (I really wanted to see 118's robot shoot!).

I do hope that FIRST will also compensate (partly or fully) the 12 adults for the cost of a trip to Manchester. Taking a trip from Florida or Nevada doesn't sound so cheap, not to mention the money lost from taking time off work...

Most team reps will be flying in Friday night and returning home Sunday night so work minimally impacted. As stated earlier, FIRST is paying for air, hotel and meals.

[EricH]

This is a step in the right direction. And a BIG one. I like the move--bring in the known "robots with problems on the field" and try for two days to either replicate the problem or find the issue, with team members present. Then the call for all the other data...

I think FIRST got the message the field sent.





Now, let's see if FIRST will send official observers to offseason events that use a FIRST field...or at least take reports of potential field issues.

[apalrd]

It could be the lack of available empty Channels for WiFi to broadcast over.

The fields currently use 5ghz 802.11N. A field has one AP with 6/7 virtual networks on the same channel set. 802.11 is also capable of running multiple networks on a single channel, although that's not terribly efficient, as the error rate will be fairly high, but that's why 802.11 needs so much bandwidth and error correction code.

However, that dosen't stop the D-link bridge from scanning all of the 2.4ghz networks it finds.

Another however, this thread is not about speculating on what happened. I am glad that they acknowledge that there is a radio/communication problem.

[Radical Pi]

The fields currently use 5ghz 802.11N. A field has one AP with 6/7 virtual networks on the same channel set. 802.11 is also capable of running multiple networks on a single channel, although that's not terribly efficient, as the error rate will be fairly high, but that's why 802.11 needs so much bandwidth and error correction code.

I was down on the floor at Einstein, and I pulled out my wifi scanner during one of the speeches. I don't remember what the 2.4 spectrum looked like, but 5GHz was completely clear with the exception of the Einstein field. However, I was surprised to find that all 6 robot networks are run on the same channel. Perhaps the router they use isn't capable of running over multiple channels.

[Gray Adams]

I am pretty sure the FMS was used at every 2012 FRC event.

They were referring to the field electronics hardware, not the FMS software (which is indeed used at all events).

[PAR_WIG1350]

wrong word choice.




Well, if look at it from the "this has been happening for years" perspective, that of a pessimist, then yes, it is the wrong choice of words. But a more optimistic person could say "at least they are doing something now instead of trying to fix it during the 2013 season", in which case it would seem like a perfectly appropriate word choice. Some one who falls somewhere in between the two groups might say that it is "ironic" for an optimist to say that they are being proactive even though this has been going on for years.

[Ether]

Well, if look at it from the "this has been happening for years" perspective, that of a pessimist, then yes, it is the wrong choice of words. But a more optimistic person could say "at least they are doing something now instead of trying to fix it during the 2013 season", in which case it would seem like a perfectly appropriate word choice. Some one who falls somewhere in between the two groups might say that it is "ironic" for an optimist to say that they are being proactive even though this has been going on for years.

I am going to use your "optimistic" definition of "proactive" next time my wife's car breaks down and she asks me why I didn't do anything about that noise she'd been complaining about for 8 weeks.

[Zebra_Fact_Man]

wrong word choice.




I can see what you mean by this, but why I selected this word, I guess, is because FIRST could have always just either blamed the individual robots or claimed it to be a fluke (or some other variation that allows them to spend no money searching for a solution) until it inevitably happened again next year.

But instead they are discrediting no claims that there was a field issue and are taking all possible steps to right the wrongs for next year. Bravo! So often do I see the wrong thing done because it is easier/more convenient/cheaper. There was a problem and they plan to fix it.

[AllenGregoryIV]

The dlinks are 5 Ghz Compatible, maybe its time to try it @ 5? There would be less interference due to the lower number of 5 Ghz compatible computers and other devices. We could run 2.4 Ghz @ home, and use 5 GHz at competitions as part of the router setup kiosk, since you are only allowed to connect via bridge or tether at competitions, it would cause no problems other than the practice fields would need Dlinks instead of the old Cisco routers.

I mean, 99% of these comm issues don't happen outside the field.

It could even be the FMS itself...

IIRC the dlinks do run on 5GHZ on the field.

[Ether]

There was a problem and they plan to fix it.

I heartily applaud FIRST for finally reacting proactively to this ongoing problem.

[Kyler Hagler]

I am excited that they are doing this. This might help find out why we blacked out during a match during Newton Qualifications as well as 1717.

[Metalcrafters]

First I have to ask to please not use this thread to hash out the same issues we have all covered else where.
We were contacted almost imediatly after the Championship to offer their support and invited us to provide any and all experiences/data. I am sure this was the same for all of the final 12.
I truely feel for all of the staff and volunteers that had to deal with the aftermath or the finals and all of the events prior. You could hear it in the voice of the gentlmen I spoke to.
FIRST has been more than professional and accommodating to all involved in the finals. If it could be extended to all that had issues, they would. Our community will benifit from the final 12 test.

FIRST does realize that we as a community has made this big part of our lives, so much so it powers us through the off season and creates conditions for great success in all we do. So My team and I are pleased with the offerings that FIRST has put on the table. Will it please all that read the results? I hope so. But for the nay sayers, this is life! be iterative!
See you next year, when we start this whole process over! Just for the fun of it!
Lucas

[Andy Baker]

Thank you to FIRST for making this a priority and taking a swift course of action. Thank you also to the 12 awesome FRC teams for accommodating and helping with this project.

Sincerely,
Andy Baker

[Pauline Tasci]

step in the right direction

[Undertones]

Hmmmmm. I hope all teams can afford to send an adult representative in. Other than that, a step in the right direction, for sure.

[Al Skierkiewicz]

Just to reiterate for those jumping to the end of the thread...
The field elements only (border and mechanical parts) were stored in Tenn. The electronics had been used at four prior events and were chosen for their proven track record at those prior events.
FIRST is taking this very seriously, from Jon and Dean all the way down. Everyone is committed to finding out exactly what happened. FIRST started this process right after Champs and has been working hard to collect data from all involved.

Thank you to all teams that are participating.

[Chris Fultz]

Hmmmmm. I hope all teams can afford to send an adult representative in. Other than that, a step in the right direction, for sure.

Based on other posts, it appears FIRST is paying the expenses for getting the robots and the people there for the weekend.

[Jared Russell]

My number one fear is that everything just works in Manchester, as it has for 95%+ of matches all season.

Here's hoping that the Einstein field fails spectacularly (and repeatably). Never thought I would have to say that sentence...

[Peter Matteson]

The field elements only (border and mechanical parts) were stored in Tenn. The electronics had been used at four prior events and were chosen for their proven track record at those prior events.

Do we know which events that field was at?

I'm asking because several of us saw consistent issues with the electronics that were on the field that went from BAE to WPI to NYC to BOS to CT. I think several teams have recoreded the data pretty well on that particular field so I'm very curious if it was the same field package.

[SillyRabbitt]

I am a mentor/coach for a rookie team and I just have one question about the final 12 headed to FIRST. How many of the robots in question that lost communication during the matches had on-board cpu's? Its just a thought that might be an issue. One more x-file! :D :D

[rsisk]

Just to reiterate for those jumping to the end of the thread...
The field elements only (border and mechanical parts) were stored in Tenn. The electronics had been used at four prior events and were chosen for their proven track record at those prior events.
FIRST is taking this very seriously, from Jon and Dean all the way down. Everyone is committed to finding out exactly what happened. FIRST started this process right after Champs and has been working hard to collect data from all involved.

Thank you to all teams that are participating.

Al, when you say the electronics were used elsewhere, could you clarify which components you are referring to, if you have the info? The three electronic components can be categorized as:

1. FMS/Stinger
2. Scoring electronics
3. Driver station electronics

[DampRobot]

This is a great step by FIRST, and I applaud its efforts to try to make sure this never happens again.

Having all 12 teams there worries me a little bit though. People have discussed "replaying" Einstein at IRI, and it almost seems like FIRST has accidentally provided this. I hope that these field tests don't devolve into an attempt by teams to try to see what would have happened if the problems had never come up.

[Francis-134]

This is a great step by FIRST, and I applaud its efforts to try to make sure this never happens again.

Having all 12 teams there worries me a little bit though. People have discussed "replaying" Einstein at IRI, and it almost seems like FIRST has accidentally provided this. I hope that these field tests don't devolve into an attempt by teams to try to see what would have happened if the problems had never come up.

I doubt that the one or two people from each team would be able to compete at anywhere near the level of the Championship, especially without any pit support, most likely the drive coach and at least one driver. I think 25, 16 and 180's well fought and earned victory is safe from controversy.

[Al Skierkiewicz]

Richard,
All non-mechanical parts, is what I have been told. That seems to fit your entire list.
I don't know what events used this package, but after testing prior to the season, most fields end up in three or four events as I remember.
This is not an attempt to replay Einstein, it is an attempt to duplicate the problems and find the cause(s) and document them. The only thing that can't be duplicated is the raging storm outside in St. Louis and the power distribution in the dome. As Jon has pointed out, everything else will be in Manchester.
I wouldn't read into this in any way. FIRST wants to get to the bottom of the problems and it seems they are doing just that. Getting 12 robots, all the team people, food and hotel, and the staff of experts to Manchester and back home is a huge expense. That should tell how committed they are.

[JaneYoung]

Having all 12 teams there worries me a little bit though. People have discussed "replaying" Einstein at IRI, and it almost seems like FIRST has accidentally provided this. I hope that these field tests don't devolve into an attempt by teams to try to see what would have happened if the problems had never come up.


My first reaction to this is, why even go there? My response to my own question is along the lines of the professional attitude and expertise that each team will bring to the field tests. Nothing will devolve on the part of these Einstein worthy teams. They have an amazing opportunity and they will take full advantage of it in order to improve the situation and help to solve the problems.

Jane

[billbo911]

...... it is an attempt to duplicate the problems and find the cause(s) and document them. The only thing that can't be duplicated is the raging storm outside in St. Louis and the power distribution in the dome. As Jon has pointed out, everything else will be in Manchester.
...

I think there is one thing they will not be able to fully replicate in Manchester, the audience and all their electronic devices.

This is just my $.02 and based on comments of others and observation.

Which matches have the most attendees? The World Finals on Einstein. Those are followed closely by the eliminations on each field.
My understanding is that each field also experienced problems as they moved into the eliminations.

More people = More devices = More potential for interference.

I'm not saying this is the root cause, but it really needs to be considered if they want to completely replicate the experience on Einstein.

[rsisk]

Keep in mind that there are many ways the investigation by FIRST can turn out:

1. They are certain the cause of the problem(s) were found and fixed 100%. Ideally this is what we are all hoping for, but chances are probably 50/50 it will occur.

2. They are confident problem(s) were found and fixed. Issues don't recur in test environment but failures do appear at IRI (for example).

3. Things were found that could be the cause of the problem(s) but due to the intermittent nature of the problem or the inability to recreate the problem in a test environment no one can be sure the problem is solved.

4. Problem(s) are caused by multiple issues combined to produce a failure. Some of the defects are solved and the problem does not reappear... until later when another combination of issues causes the failure to recur.

5. Nothing is found

The nature of intermittent problems in a complex system means you will not likely be 100% confident problem(s) have resolved. The best you can hope for is moving the confidence level more towards the 100% level.

By FIRST taking the effort to conduct a significant dive into the problem, including attempting to recreate the environment as much as possible helps tremendously in letting us know that FIRST recognizes there is a problem and they are making an effort to resolve it. Results in the 1-4 category above will help move the confidence level in the field towards the 100% level (some more than others). If category 5 happens, well, the confidence level will probably stay where it is at.

Kudos to FIRST and all involved in the investigation for taking the time, effort, resources, and dollars necessary to deal with this issue.

[Kims Robot]

Having all 12 teams there worries me a little bit though. People have discussed "replaying" Einstein at IRI, and it almost seems like FIRST has accidentally provided this. I hope that these field tests don't devolve into an attempt by teams to try to see what would have happened if the problems had never come up.
I'm fairly certain there is zero chance of this... even if each team was allowed to bring more than 1 or 2 people. The poise with which all of the teams on Einstein handled the situation was exactly the definition of GP. By any of our standards, no one would have been surprised if one of the teams drivers or coaches threw up their hands in frustration or stomped around the edge of the field. Yet by everything I saw (we had direct view of the red side of the field), everyone remained calm and cool, and I even watched video showing the losing teams clapping with smiling faces. How they pulled that off is beyond me, but true evidence that they are Einstein caliber teams. I don't think any of those teams will be out to change our minds, and I think all of them have earnest interest in helping us all solve the issues and end up with a system that REALLY works so we can all play at our fullest potentials next year.

[Hjelstrom]

Keep in mind that there are many ways the investigation by FIRST can turn out:

1. They are certain the cause of the problem(s) were found and fixed 100%. Ideally this is what we are all hoping for, but chances are probably 50/50 it will occur.

2. They are confident problem(s) were found and fixed. Issues don't recur in test environment but failures do appear at IRI (for example).

3. Things were found that could be the cause of the problem(s) but due to the intermittent nature of the problem or the inability to recreate the problem in a test environment no one can be sure the problem is solved.

4. Problem(s) are caused by multiple issues combined to produce a failure. Some of the defects are solved and the problem does not reappear... until later when another combination of issues causes the failure to recur.

5. Nothing is found

The nature of intermittent problems in a complex system means you will not likely be 100% confident problem(s) have resolved. The best you can hope for is moving the confidence level more towards the 100% level.

By FIRST taking the effort to conduct a significant dive into the problem, including attempting to recreate the environment as much as possible helps tremendously in letting us know that FIRST recognizes there is a problem and they are making an effort to resolve it. Results in the 1-4 category above will help move the confidence level in the field towards the 100% level (some more than others). If category 5 happens, well, the confidence level will probably stay where it is at.

Kudos to FIRST and all involved in the investigation for taking the time, effort, resources, and dollars necessary to deal with this issue.

This is a great way to describe the problem in front of us. The fact that First is going "all out" to reproduce the problem will help improve the chances of a positive outcome.

[Taylor1023]

FIRST is the only organization I know of that will go as far as to ask 12 teams to come from around the country to test out a (possibly) faulty playing field to make sure that the game was fair. This is truly gracious professionalism and coopertition at its finest.

[CalTran]

FIRST is the only organization I know of that will go as far as to ask 12 teams to come from around the country

Around the world. The Eh-team is Canadian. Simbot Facebook said Simbot Jordan was ever so close to home before he got his ticket to FIRST HQ. I dunno where ATA or OP Robotics are, but I'm sure they're meeting their buddy Simbot Jordan at FIRST too

[Al Skierkiewicz]

I am a mentor/coach for a rookie team and I just have one question about the final 12 headed to FIRST. How many of the robots in question that lost communication during the matches had on-board cpu's? Its just a thought that might be an issue. One more x-file! :D :D

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

[cpeister]

Around the world. The Eh-team is Canadian. Simbot Facebook said Simbot Jordan was ever so close to home before he got his ticket to FIRST HQ. I dunno where ATA or OP Robotics are, but I'm sure they're meeting their buddy Simbot Jordan at FIRST too

2056 is from Stoney Creek ON, just up the road from 1114. 4334 is from Calgary AB.

[Taylor]

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

Our "losing communications" moments at QCR involved 3 matches of a working hybrid mode, then as soon as teleop started, no comms, no code. One match, about 3 seconds of hybrid mode, then no comms, no code. One match - as soon as the match began, no comms, no code. Four of the five matches, a connection was reestablished with about :10 left in the match; the last one, we never had connectivity during the match. These issues appeared at different stations and different alliances. When we reset the router firmware from v1.4 to v1.2 it appeared to fix the problems (the last two matches, we saw no communication issues). We had no comm issues at BMR, during practice at QCR, or the first half of Friday at QCR. We had no extra onboard computer.
As Al said, the "losing communications" catchall really envelops a lot of symptoms and issues, and ours -seems to- have been traced back to an individual robot issue.
It is a tangled web indeed, and I do not envy those charged with unraveling it. As has been said before, I hope a bumpercrop of identifiable and repeatable issues pop up in June.

Sorry to hijack the Einstein thread with a regional vignette, but I felt it was part of the greater picture.

[Al Skierkiewicz]

When we reset the router firmware from v1.4 to v1.2 it appeared to fix the problems (the last two matches, we saw no communication issues). We had no comm issues at BMR, during practice at QCR, or the first half of Friday at QCR.
Anybody else experience this? Taylor is it possible that the firmware was corrupt? Had you tried re-flashing the router with the v1.4? Not to hijack the thread, but all of you can PM me with this info. Thanks.

[Holtzman]

Anybody else experience this? Taylor is it possible that the firmware was corrupt? Had you tried re-flashing the router with the v1.4? Not to hijack the thread, but all of you can PM me with this info. Thanks.

At GTR-W, all the teams with firmware 1.4 had difficulties connecting to the field whenever there were more than 3 robots on the field at a time. We were one of about a half a dozen teams that had to downgrade our router firmware to 1.21. The problem was first noticed on Thursday afternoon when we started getting more than 3 robots at a time for practice matches. Anytime a robot with Firmware 1.4 went out on the field and powered on, ping times for all teams would spike, and robots would start randomly dropping out and reconnecting. The running theory was that the combination of a large number of access points, and the something in the 1.4 firmware was causing communication to drop in and out. By late Thursday afternoon, everybody downgraded to 1.21, and the event ran smoothly.

Figuring out that it was specific robots that was causing the problem was tricky. We couldn't figure out how something specific to our robot was causing so many difficulties for all the other teams on the field, especially since we had already attended two other regionals, and run without a single issue at either. The FTA asked all the teams that seemed to be causing the field difficulties to check their firmware, and we realized it was the one thing we all had in common. Thinking back, I can't recall if it was something that inspectors checked that at the Championship. Does anyone else remember being asked router firmware at the Championship?

It was one of the things that each Einstein team was asked in the questionnaire we filled out.

[Chris Fultz]

We had issues at SMR with 1.4 and stopped in 3 matches.

We made other changes also to reduce the CPU demand, but also downgraded to 1.21 and did not have problems at BMR or CHP.

[JohnBoucher]

This was noted back in March
http://www.chiefdelphi.com/forums/showpost.php?p=1146291&postcount=35

[class1234567]

I dont know if im saying this right but when every my team (1985) and 1986 where on the same field at the same time the FMS whould shut down and this happened at GKC at STL regionals so i was wondering is FIRST looking into this as well

[JB987]

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

Al,
Our robot has an on board cpu (Panda Board) for our on board Kinect and we never had any problems until Einstein.

[Al Skierkiewicz]

Joe,
The CPU likely was not the problem but anything is possible. How did you interface to the control system?

[Alan Anderson]

Sometime after Week 4 of regional competitons, CSAs were asked to check the WPA kiosk logs to identify teams with D-Link firmware version 1.40 and to watch for issues. Maybe not all of them got the memo.

[steelerborn]

Just a thought, but I couldn't help but notice that there was a lot of spare time between when the divisions finished playing and the start of Einstein. It would be a good idea to have 6 bots from the divisions, that did not make it to Einstein, play a few rounds just to test the field systems? You could have a raffle to see which teams make it to the pre-Einstein "just for fun" event, this way the people in the stands have something to watch (although the paper airplanes were a good distraction :D), and also if the field does have any issues you would be able to catch them before the first matches start. And you can just use leftover basketballs (or some other game piece for next year) from the divisions, so Einstein has a fresh field still.

[Tom Bottiglieri]

At CMP, we ran laggy controls (robot responded 2 seconds after operator input, tons of dropped packets, long trip times) on 1.4. Downgraded to 1.21 and all was well.

[EricVanWyk]

I dont know if im saying this right but when every my team (1985) and 1986 where on the same field at the same time the FMS whould shut down and this happened at GKC at STL regionals so i was wondering is FIRST looking into this as well

At CMP, we ran laggy controls (robot responded 2 seconds after operator input, tons of dropped packets, long trip times) on 1.4. Downgraded to 1.21 and all was well.

These should also be reported to the email address in Jon's letter.

[Robogineer1649]

Just a thought, but I couldn't help but notice that there was a lot of spare time between when the divisions finished playing and the start of Einstein. It would be a good idea to have 6 bots from the divisions, that did not make it to Einstein, play a few rounds just to test the field systems? You could have a raffle to see which teams make it to the pre-Einstein "just for fun" event, this way the people in the stands have something to watch (although the paper airplanes were a good distraction :D), and also if the field does have any issues you would be able to catch them before the first matches start. And you can just use leftover basketballs (or some other game piece for next year) from the divisions, so Einstein has a fresh field still.

I like your idea but potentially one of the robots could damage the field not just the game pieces, but i noticed that the bridges tended to become damaged from certain bridge manipulators. Wheels also tended to scrape up the plastic covering that the bridges had on them.

[JB987]

Joe,
The CPU likely was not the problem but anything is possible. How did you interface to the control system?

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

[trilogy2826]

One additional data point to add to the pile that I don't believe was mentioned in any of this years previous "Communincations Issues" posts:

Specs:
Bridge Firmware: 1.21
DS Laptop: HP Elite
Axis camera
Vision targeting through separate PC connected through ethernet into bridge
C++
4 port cRio

When 2826 arrived on Archimedes for the first practice match:
- We noticed an extremely high, relatively constant trip time (> 150ms) through the diagnostic tab on the DS.
- Our FMS connection indicator would rapidly turn off and on and our axis camera feed was extremely lagged.
- For the first 30 seconds, we were pretty much dead in the water.
- We "jiggled" the ethernet connection and check CPU and network utilization, and the numbers were low and consistent with times where no errors ocurred
- We then unplugged and re-plugged our ethernet connection to the DS laptop mid match. For an unknown reason, everything suddenly worked perfectly. No latency, no lag and full control
-Afterward, the FTA mentioned that our trip time suddenly decreased per their diagnostics, so that was consistent

For Practice Match 2:
- We had the same rapid disconnection issues as match 1 as soon as we connected the ethernet cable to the DS laptop
- The FTA came to us before the match was started and we tried unplugging and re-plugging the ethernet cable. This did not fix the issue, same symptoms
- The FTA's curiousity was piqued, so he allowed a bit of troubleshooting
- We could ping the bot with consistently high latency
- We disabled and enabled the DS laptop network adapter - No fix
- The FMS was rebooted - No Fix
- Robot was rebooted - No Fix
-DS laptop OS (windows 7, obviously) was restarted - At this point we had a completely new behavior in that we could ping the robot with low latency, but the FMS would not show connection. Quote from FTA "What the heck?! I've never seen this before"
-We started the match anyway and had no control or feedback from/to the bot
-About 10 seconds into the match, we disconnected and reconnected the ethernet cable and viola! Everything worked again.

*At this time, we had a consistent set of failures and symptoms, but no consistent correction method. We were starting to freak out...

For Practice Match 3:
- Immediately prior to approaching the Alliance station, we rebooted the DS laptop
-We waited for at least 1 minute for the robot and bridge to boot
-We waited for the DS and dashboard applications to fully launch.
-We confirmed that "2826" was established on the LCD board above our alliance station
-We plugged in the ethernet cable into the DS laptop and got immediate connection to the robot and FMS. Little to no latency (<2-3 ms on average)
-we ran the match with no glitches or issues

At this point we knew our DS laptop was somehow perpetuating the issue, if not directly causing it. We ran the process described in 3 more practice matches, all Quals and elims with absolutely no issues.

We know that we did not solve anything, but hopefully this process can shed some light on other noted issues.

[slijin]

At this point we knew our DS laptop was somehow perpetuating the issue, if not directly causing it. We ran the process described in 3 more practice matches, all Quals and elims with absolutely no issues.

On a similar point, we've also seen the DS software itself to be faulty, where only by closing and reopening the DS could a connection to either the robot or Cypress board be reestablished.

[Thad House]

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

I also have heard the 118 uses a BeagleBone to process their camera.

[DMetalKong]

I saw this posted in another thread (http://www.chiefdelphi.com/forums/showthread.php?t=106094). I have not seen this specific issue mentioned in this thread so far, but it seemed relevant so I thought I would bring it up.

We used an Arduino Mega with an Ethernet shield this year for the RPM on the shooter and the angle of our ball launch. Using the Ethernet wasn't too hard. But, the cRio had something odd in the robot code where if you started sending packets, but there was nothing actively reading them, that the robot would eventually go offline. We were using UDP on the local link as a directed unicast to keep packets from going across the link to the driver station.

It's definitely related to the robot code because I've been using VxWorks for over 25 years and I've never encountered anything like this failure in the main OS. Nonetheless, if you have a thread reading the packets all of the time, then everything works well. I'll be doing some tests as soon as I get access to a cRio again to see how much data the cRio can absorb via the Ethernet during normal operation before the driver station packets start getting dropped.

[linuxboy]

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

[JB987]

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

Yes.

[Hjelstrom]

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

Ok, forgive me if I describe anything incorrectly, this was our team's first experiment with writing our own network communications code (among many other things this year). We have a Pandaboard on the robot and it has an ethernet cable connecting it to the bridge. We use TCP/IP packets and the Pandaboard acts as a listening server and the cRio as a client (C++). Packets are only sent from the Pandaboard to the cRio when the cRio sends a packet requesting information. There should be no extraneous packets being sent and its all over the wire, not wireless. We've used it this way in every match all year at two regionals and at worlds. The only problem we had was one match in Las Vegas where our Pandaboard did not power up and the cRio waits up to 30s trying to establish a connection. We know the behavior of that problem and that is not what happened on Einstein. We also had one match in Curie where we did not plug our robot battery all the way in. So we had no "unknown" problems even in practice matches all year until Einstein. I don't know if any of us ever looked at what firmware our bridge is running.

I think what Joe was referring to was that we also went to extra effort this year to minimize the data we were sending from the cRio to the driver station. We set up a toggle button on the driver station controls that enables and disables data sending (disabled by default). The reason for this is that we noticed that the driver station quickly becomes very "laggy" when we send data to it (this is using SmartDashboard and NetworkTables). First we tried to solve this by upgrading the laptop but it still happened. So we worked around the problem by not sending data unless we really needed to "debug". Even with our data sending enabled, we were using a timer to drastically limit how often we send data. It seems very odd to me that the small amount of data being sent from the robot to the DS could make any difference in the DS performance.

We stuck with SmartDashboard because for another part of our tele-op control system, the flexibility of being able to send a small amount of data from the SmartDashboard to the robot was so useful (i.e. the functionality of NetworkTables). It was worth all of the trouble we had to go through and all of the workarounds like restarting our DS before each match.

[JB987]

Ok, forgive me if I describe anything incorrectly, this was our team's first experiment with writing our own network communications code (among many other things this year). We have a Pandaboard on the robot and it has an ethernet cable connecting it to the bridge. We use TCP/IP packets and the Pandaboard acts as a listening server and the cRio as a client (C++). Packets are only sent from the Pandaboard to the cRio when the cRio sends a packet requesting information. There should be no extraneous packets being sent and its all over the wire, not wireless. We've used it this way in every match all year at two regionals and at worlds. The only problem we had was one match in Las Vegas where our Pandaboard did not power up and the cRio waits up to 30s trying to establish a connection. We know the behavior of that problem and that is not what happened on Einstein. We also had one match in Curie where we did not plug our robot battery all the way in. So we had no "unknown" problems even in practice matches all year until Einstein. I don't know if any of us ever looked at what firmware our bridge is running.

I think what Joe was referring to was that we also went to extra effort this year to minimize the data we were sending from the cRio to the driver station. We set up a toggle button on the driver station controls that enables and disables data sending (disabled by default). The reason for this is that we noticed that the driver station quickly becomes very "laggy" when we send data to it (this is using SmartDashboard and NetworkTables). First we tried to solve this by upgrading the laptop but it still happened. So we worked around the problem by not sending data unless we really needed to "debug". Even with our data sending enabled, we were using a timer to drastically limit how often we send data. It seems very odd to me that the small amount of data being sent from the robot to the DS could make any difference in the DS performance.

We stuck with SmartDashboard because for another part of our tele-op control system, the flexibility of being able to send a small amount of data from the SmartDashboard to the robot was so useful (i.e. the functionality of NetworkTables). It was worth all of the trouble we had to go through and all of the workarounds like restarting our DS before each match.

Thanks Greg...so much better than my second hand translation;)

[BHOP]

Thank you FIRST!

I think they should also figure out a way to get (or simulate) a couple thousand (10s of thousands?) WiFi receiving/transmitting devices like cell phones, laptops, etc into their test.

It seemed there were more problems as more people started watching matches, in the elims and then accumulating to einstein where everyone was surroundinng one field with their wireless devices.

or maybe FIRST should just license some white space spectrum :ahh:

[Al Skierkiewicz]

There were actually very few on 2.4 GHz. Maybe 4 or 5. The field runs on 5 GHz and there are very few phones that can transmit that at this time.

[Libby K]

A great big step in the right direction.
:)
Thank you, FIRST.

[DonRotolo]

The only thing that can't be duplicated is the raging storm outside in St. Louis and the power distribution in the dome.If FIRST was really committed, they'd fly the dome in, and use one of Dean's weather-controlling machines. :p
Thank you FIRST!

I think they should also figure out a way to get (or simulate) a couple thousand (10s of thousands?) WiFi receiving/transmitting devices like cell phones, laptops, etc into their test.


or maybe FIRST should just license some white space spectrum
RF interference can easily be simulated. Ten thousand devices look like a high level of RF noise to any other device, so just broadcasting some wideband RF will effectively duplicate this condition. This is a common practice in the industry.

Maybe instead FIRST should require every team member and volunteer to get their Amateur Radio license, and we can then just use the Ham bands :p

In all seriousness: FIRST is tossing significant resources at the issue, and I for one am very comfortable with the process they are following so far. Kudos for good communications, too.

[EricS-Team180]

Maybe instead FIRST should require every team member and volunteer to get their Amateur Radio license,
.
Funny you should say that, Don. Bryce, who's attending for S.P.A.M. just so happens to have his Ham license :D

FRC 180 is totally behind these efforts by F.I.R.S.T , and we sure hope that the testing will yield insight into what I'll just call "the comms problem". It has been vexing us for awhile, now.

[Andy Brockway]

.
-DS laptop OS (windows 7, obviously)

We are still running the 2009 classmate with XP. I have installed every update. None of the thumb drives from FIRST have worked to reflash the drive. The first year FIRST sent me a new one but still I could not reflash.

My team has had no connectivity issues. We are using the older bridge and firmware.

[Rangel(kf7fdb)]

Maybe instead FIRST should require every team member and volunteer to get their Amateur Radio license, and we can then just use the Ham bands :p


Team 842 is way on top of that. Nearly every member of our team has a ham radio license. :D

[kenavt]

I also have heard the 118 uses a BeagleBone to process their camera.

I believe this is correct.

http://www.chiefdelphi.com/forums/showpost.php?p=1166193&postcount=9
http://www.chiefdelphi.com/forums/showpost.php?p=1138572&postcount=19

To add to the Driver Station talk, for the first year we had an '09 Classmate that over time became more and more faulty, and is currently stuck in a rebooting loop. This may have been to due to a lack of SSD space, as numerous programs were loaded, including LabVIEW.

The replacement we used, a small Asus netbook, also succumbed to technical issues (I believe they were related to GRUB and dual-booting Ubuntu alongside Windows 7) somewhat quicker.

The next laptop we used, for all of the World Championship, was my own personal Acer. I'm running Win7 HP 64-bit, with only two user accounts - my personal one and 'Driver'. There were no problems whatsoever.

The only time we had major latency issues on the field was during an early match at Kettering in Week 1. However, this appeared to be due to our custom Dashboard on the DS sending ~50 packets a second to the cRIO. Once this code was removed, then there were no latency issues that I know of.

[JohnBoucher]

If I am not mistaken this weekend is the Manchester test of the Einstein robots and the Einstein field. After some initial announcements, FIRST has been very quiet about this.

Attn FIRST: The CD community and the FIRST community are thankful that you are working hard to understand what happened on Einstein at champs. In order to restore the confidence of the FIRST community you need to communicate to us what is being done. We initially received a few communications, but there has been none leading up to this weekends test sessions. Please make this process more transparent. We need information. We do not expect that you will publish final results after this weekend, but we need to know what is being done. Please keep us in the loop.

[Nick Lawrence]

I'm terrified that they've spent all this time, money and the like...

... and find nothing.

I too would like them to be more transparent about this process.

-Nick

[rsisk]

...even if nothing is found. Let us know

[Jon Stratis]

If I am not mistaken this weekend is the Manchester test of the Einstein robots and the Einstein field. After some initial announcements, FIRST has been very quiet about this.

Attn FIRST: The CD community and the FIRST community are thankful that you are working hard to understand what happened on Einstein at champs. In order to restore the confidence of the FIRST community you need to communicate to us what is being done. We initially received a few communications, but there has been none leading up to this weekends test sessions. Please make this process more transparent. We need information. We do not expect that you will publish final results after this weekend, but we need to know what is being done. Please keep us in the loop.

Probably the most damaging thing FIRST could do right now would be to release any information that ends up not being 100% accurate. What more can they tell us other than they're investigating and having the teams come out? What would you and everyone else here think if they said "We've determined the issue is not with the wifi routers", and a month form now they change their minds?

There's a lot they can do to investigate the issue and try to reproduce it themselves... but until they can reproduce it with team robots and setup, they can't be 100% sure they've found the issue. They need to reproduce it with the teams, fix it with the teams, AND prove that the fix actually worked before they can be sure of anything. And with an intermittent issue like this, proving it can be quite a chore.

Half of my job is to triage issues in our production system... Sometimes, it's easy. Sometimes, it's an intermittent issue that's almost impossible to reproduce reliably. Lets say you run through a test for the issue, and you encounter it only 3 times out of 10 runs. You think you understand it and apply a fix. To prove that it's actually fixed, how many times do you need to run your test? How reliable is your test if it isn't run on a production-equivalent environment? Debugging issues is difficult enough when your users interact with your system through a clearly defined interface (like a website). Debugging them when your users are writing their own custom code that interfaces with your system has to be a nightmare.

[dcherba]

I have controlled lots of items through ham radio and it still comes down to design to get reliable operation. By the way our team was founded by the ARGYL youth ham radio club in Lowell
Team 3234 Red Arrows. Currently about 1/3 of our students have their ham license.

[JohnBoucher]

Probably the most damaging thing FIRST could do right now would be to release any information that ends up not being 100% accurate.
I appreciate this, but there is a lot that can be shared without specifics. The CD community has been very vocal on opening up the lines of communication and having FIRST be more transparent. They have asked for our input on the field issues. I believe the communication can be in both directions.

[Astrokid248]

One thing that they could share with us is their testing procedure. It would be nice to know, and it could be used for educational purposes. It could also allow us to suggest further testing based on what FIRST has already looked for.

[Tom Line]

One thing that they could share with us is their testing procedure. It would be nice to know, and it could be used for educational purposes. It could also allow us to suggest further testing based on what FIRST has already looked for.

Wouldn't that be remarkable? If one of their engineers actually logged on here and submitted their test plan, then posted results (but NO conclusions)?

Folks could see real-time problem solving at work, and real-time engineering.

[cgmv123]

One thing that they could share with us is their testing procedure. It would be nice to know, and it could be used for educational purposes. It could also allow us to suggest further testing based on what FIRST has already looked for.

They should live stream it.

[Alexa Stott]

Wouldn't that be remarkable? If one of their engineers actually logged on here and submitted their test plan, then posted results (but NO conclusions)?

Folks could see real-time problem solving at work, and real-time engineering.

But people on here are going to criticize them regardless of what they do.

[wilsonmw04]

But people on here are going to criticize them regardless of what they do.

Quoted for truth. There is a faction of the CD community that will hammer FIRST no matter what they say or do because it will never be enough. A lot of them are mentors and leaders of teams. Isn't part of any engineering project learning to deal with circumstances beyond our control? Isn't it important that we model professional behavior instead of gnashing teeth and hopping on the "FIRST fails" bandwagon?

FIRST has said they are looking into the problem. I have a feeling that the teams involved will be more than happy to communicate what they have learned from the weekend’s events. Let’s show a bit of patients and wait until after the tests have been completed. I guess a sense of entitlement isn’t limited to the students we teach…

[meaubry]

I do not believe this has to do with entitlement, but rather having a vested interest - and hoping to hear back with what was done & results.
There will always be folks that look at things with optimism and others with pessimism - that is human nature. This site is no different.
There are many people that have posted that truely want to help, while others are simply curious.
But, I think alot of folks involved in the FIRST experience (especially this year) just want to get feedback, relative to how the process of trouble shooting and root causing was done, and the results from the testing done with the teams.
Perhaps, the FIRST folks just need a bit more time to pull the report together - so, yes - patience is a great suggestion.
For now.

[pfreivald]

Giving them a chance to do a real investigation and put forth proper, well-founded, data-based conclusions. That might take a week. Or six. Or twenty.

We know they're working on it. We don't need our hands held every step of the way. I, for one, would rather they spend their effort and energy getting it right, rather than keeping us informed so we can pick apart and micromanage their processes.

[JaneYoung]

It's a good time to say, thank you, to everyone involved.

There's a big difference between transparency and a fish bowl. These people need the freedom to think and to work together without the pressure of a reality show atmosphere that the CD community could help to create if they were more transparent with the process.

They need this time to work and all they need from us is our support. They know they've got that.

Jane

[Astrokid248]

There's a big difference between transparency and a fish bowl. These people need the freedom to think and to work together without the pressure of a reality show atmosphere that the CD community could help to create if they were more transparent with the process.

Agreed; live-streaming, or even posting pictures, might be going a bit too far. Most people I know get weird on camera, and that won't help these guys solve the issue. I still want to know their process, but anything more than a few paragraphs in an email is asking for way too much.

I think they are aware of how much this means to us, and how difficult it is to wait on their results. But I think they also have to consider what's best for the troubleshooting process. That's why they are spending only one weekend on it. It's Thursday, Monday is the soonest we can expect a summary of the tests (but not the conclusions, I give them another week for that). We've done all the speculation we can do (some of us with less knowledge than others, ahem, me); let's rest our minds and keyboards and wait.

[Peter Matteson]

It's a good time to say, thank you, to everyone involved.

There's a big difference between transparency and a fish bowl. These people need the freedom to think and to work together without the pressure of a reality show atmosphere that the CD community could help to create if they were more transparent with the process.

They need this time to work and all they need from us is our support. They know they've got that.

Jane

Jane,
I think the issue is that many of the members of these boards do Root Cause Corrective Action (RCCA) in our everyday jobs. This is large base of expertise that could hel;p out including mentors who work for suppliers that make the components in question. I know I would feel better if they solicited peer review of their plan because there a lot of people who think they know how to properly do this type of work but don't really go in depth on it, or understand how to apply the ACE/Six Sigma processes.

I agree that they should not do this in a bubble, I'm just not sure that FIRST has demonstrated they have the most capable people involved in the process, and if they had asked for support many of us could have aided in the preliminary fault tree or other preliminary trouble shooting to set up the test plan. I hope that representatives from NI and CISCO were at least consulted for formulating the test plan.

[Kristian Calhoun]

I hope that representatives from NI and CISCO were at least consulted for formulating the test plan.

I believe they have, as several will be present at the testing.

[JaneYoung]

Jane,
I think the issue is that many of the members of these boards do Root Cause Corrective Action (RCCA) in our everyday jobs. This is large base of expertise that could hel;p out including mentors who work for suppliers that make the components in question. I know I would feel better if they solicited peer review of their plan because there a lot of people who think they know how to properly do this type of work but don't really go in depth on it, or understand how to apply the ACE/Six Sigma processes.

I agree that they should not do this in a bubble, I'm just not sure that FIRST has demonstrated they have the most capable people involved in the process, and if they had asked for support many of us could have aided in the preliminary fault tree or other preliminary trouble shooting to set up the test plan. I hope that representatives from NI and CISCO were at least consulted for formulating the test plan.

Well, the question falls to how FIRST solicited help, including peer review. And, for many us, that answer is - we don't know.

I understand the concern.

Jane

[Jon Stratis]

Jane,
I think the issue is that many of the members of these boards do Root Cause Corrective Action (RCCA) in our everyday jobs. This is large base of expertise that could hel;p out including mentors who work for suppliers that make the components in question. I know I would feel better if they solicited peer review of their plan because there a lot of people who think they know how to properly do this type of work but don't really go in depth on it, or understand how to apply the ACE/Six Sigma processes.

I agree that they should not do this in a bubble, I'm just not sure that FIRST has demonstrated they have the most capable people involved in the process, and if they had asked for support many of us could have aided in the preliminary fault tree or other preliminary trouble shooting to set up the test plan. I hope that representatives from NI and CISCO were at least consulted for formulating the test plan.

FIRST didn't send out an e-mail blast asking for help, true... but they contacted those they needed to with the appropriate expertise to look into the issue. Rather than get a large number of people offering advice on how to proceed, it's important to get the right people in on the project offering advice. As a community of team members and mentors, we're being represented by those going for the Einstein testing. If they need an expert on the cRio, they'll contact engineers at NI directly, and that's a communication we won't be a part of. Likewise for all of the other aspects of the control system.

You know how the FIRST community is... if they asked us as a broad community for help, they would end up spending more time reading, organizing, and replying to our "help" than they would actually addressing the problem. Too many cooks in the kitchen, as the saying goes.

When I'm debugging a problem in our production environment (which is part of my job), I don't get our entire development team into a room to look at it. I perform initial triage, then pull in experts with specific knowledge that is directly applicable to the issue at hand. More people would only add confusion, delays, and arguments about the best way to tackle the issue.

[JohnBoucher]

It's not about entitlement or putting FIRST into a bubble so we all can debate their every move. From the May 15th email. Please allow me to update you on our examination of the Championship final rounds. Just keep us in the loop, That's all I'm asking.

I think that no communication sends the wrong message.

[Peter Matteson]

FIRST didn't send out an e-mail blast asking for help, true... but they contacted those they needed to with the appropriate expertise to look into the issue. Rather than get a large number of people offering advice on how to proceed, it's important to get the right people in on the project offering advice. As a community of team members and mentors, we're being represented by those going for the Einstein testing. If they need an expert on the cRio, they'll contact engineers at NI directly, and that's a communication we won't be a part of. Likewise for all of the other aspects of the control system.

I didn't expect FIRST to solicit help from everyone,because yes this would be counter productive. I would just ask targed teams/mentors to aid in the investigation and mention this in the updates.

I expected the flow of information to mention experts from the vendors involved being called in, and maybe some senior/well known mentors to be involved. Kristian confirmed that at least the suppliers are involved which makes me more comfortable than I was before.

[Nick Lawrence]

Just got this email from FIRST.

ear FRC Teams:



Thank you for your continued patience, support, and submissions as we conduct the evaluation of Einstein from Championship. This examination and the ultimate solutions remain the top priority for the FRC team. Here are the most up-to-date actions and information.



The final twelve teams who competed on Einstein, including 1-2 representatives and the team robot, will participate in the evaluation this upcoming weekend (June 9-10) in New Hampshire.



FRC is recreating a full competition field, utilizing the same electronics present at Championship, on which to conduct the testing. We are duplicating all possible conditions to match the St. Louis environment. While it can never be a perfect replication, FIRST has matched each possible condition within its control, including using the same Championship production company to provide the same lighting conditions.



In addition to FIRST staff and team representatives, 18 additional key FIRST volunteers and outside experts from across the U.S. will be in attendance to collect data, form hypotheses, and analyze results.



We are deeply grateful to the twelve final Einstein teams for their whole-hearted participation and support. At every juncture, they have openly shared information and been extremely forthcoming and helpful – true models of Gracious Professionalism. We also are thankful to our community for sharing their 2012 season experiences, theories, and advice. Thank you for your continued patience as we move through this important process.



We believe this upcoming weekend will be tremendously instructive and productive.



Sincerely,



Jon

Hmmm.

-Nick

[Mr. Van]

Please forgive me if this idea has been posted before - I don't have time to look through all 8 pages of this (and many other) threads about the issue. I'm not an expert, but I just thought of a possible aspect of the problem that probably can't be replicated in New Hampshire. I want to at least mention it.

Is is possible that as the finals were happening, and the storm was building, numerous - as in hundreds - of people were using their smart phones to check the weather, contact folks to see if they are ok, etc.?
From what I understand, many of these phones will be looking for wireless access points - like robots and the field. (I've noticed that my phone picks up our robot...) of course, they can't connect to the robot or the field, but many of the phones are at least pinging them - trying to make a connection - using bandwidth.

I think this could explain why the problem existed so prominently on Einstein and wasn't an issue in previous years.
1. Many more spectators - more smart phones
2. Storm - more smart phones in use than normal circumstances
3. More smart phones in the population in general.

Once again, I'm not an expert, and I'm sorry if this sort of thing has been mentioned before. I hope that if I mention it here and it is useful, it will get to those who need it.

I hope they figure this out!

- Mr. Van
Coach, Robodox

[Andy Baker]

Jane,
I think the issue is that many of the members of these boards do Root Cause Corrective Action (RCCA) in our everyday jobs. This is large base of expertise that could hel;p out including mentors who work for suppliers that make the components in question. I know I would feel better if they solicited peer review of their plan because there a lot of people who think they know how to properly do this type of work but don't really go in depth on it, or understand how to apply the ACE/Six Sigma processes.
...

From what I heard, they have done this. 5-why, Six Sigma, Root Cause experts have been brought in to help with this. Some are FIRST experts, some are new eyes to the situation.

I'm excited to hear how things go this weekend, and wish everyone the best of luck and good fortune to find out the most they can.

Sincerely,
Andy Baker

[JohnBoucher]

I think that that was a great email from FIRST explaining where they are in the process. That's all we need to know.

FIRST needs to know that the CD community is interested in the process and the results.

[Al Skierkiewicz]

It is important to remind everyone that many First Staffers, GDC and Key Volunteers visit CD often, sometimes daily. Although many do not choose to post, they are reading what you all have to say. It is easy to see that First is taking this very seriously from that facts outlined in Jon's email. Read the letter and think about the investment that is being made by First to insure that everyone (your team) has a great experience in the future.

[Fletch1373]

Is is possible that as the finals were happening, and the storm was building, numerous - as in hundreds - of people were using their smart phones to check the weather, contact folks to see if they are ok, etc.?
From what I understand, many of these phones will be looking for wireless access points - like robots and the field.

Yes, there was a large number of phones in one area, many of which I'm sure were using wireless. The robots, however, connect to the field on the 5GHz band, as opposed to the 2.4GHz band that most devices use(I don't think I've ever seen a phone that can connect on 5GHz)

I've noticed that my phone picks up our robot...

What you're seeing is actually the Wireless SSID's that the field is broadcasting, not the robots themselves. You're unable to connect because of the WPA2 encryption that every team has to put on their robot's radios(via the radio kiosk) before they can compete. The field is configured with those encryption keys and any device connecting to it must have it as well.

[Peter Matteson]

From what I heard, they have done this. 5-why, Six Sigma, Root Cause experts have been brought in to help with this. Some are FIRST experts, some are new eyes to the situation.

I'm excited to hear how things go this weekend, and wish everyone the best of luck and good fortune to find out the most they can.

Sincerely,
Andy Baker

This was all I wanted to hear. The letter makes me feel better that they had the right people involved and clearly were addressing this thread.

I don't want to go through stories of RCCA investigation gone wrong here but trust me my concern comes from experieince where I have had to clean up bad investigations from well intentioned people so I'm always a little sensetive to these types of issues.

[linuxboy]

Yes, there was a large number of phones in one area, many of which I'm sure were using wireless. The robots, however, connect to the field on the 5GHz band, as opposed to the 2.4GHz band that most devices use(I don't think I've ever seen a phone that can connect on 5GHz)



What you're seeing is actually the Wireless SSID's that the field is broadcasting, not the robots themselves. You're unable to connect because of the WPA2 encryption that every team has to put on their robot's radios(via the radio kiosk) before they can compete. The field is configured with those encryption keys and any device connecting to it must have it as well.

The SSID is broadcast on the same frequency as the network it represents, so I imagine his phone is 5GHz capable, but as stated, this is the exception, not the rule.

[Rosiebotboss]

From what I heard, they have done this. 5-why, Six Sigma, Root Cause experts have been brought in to help with this. Some are FIRST experts, some are new eyes to the situation.

I'm excited to hear how things go this weekend, and wish everyone the best of luck and good fortune to find out the most they can.

Sincerely,
Andy Baker

Not to belabor this, but I am at FIRST HQ now for the FLL/RD/FTC partners conferences and had the opportunity to talk face to face with Bill Miller last night at dinner. (And yes, he reads ALL these posts and is VERY AWARE of the conversation ongoing here on CD)

My point is this: FIRST is doing all the right things. All the key FTAs are here, all the key Vols are here, OUTSIDE engineers that have nothing to do with FIRST are here, (some have been here all week) and the teams are here. (or coming in today) along with FIRST Staff.

Let them do there job and dive in. I am as anxious as the next guy to find out what went wrong. It will take time.

[CalTran]

My point is this: FIRST is doing all the right things. All the key FTAs are here, all the key Vols are here, OUTSIDE engineers that have nothing to do with FIRST are here, (some have been here all week) and the teams are here. (or coming in today) along with FIRST Staff.

Let them do there job and dive in. I am as anxious as the next guy to find out what went wrong. It will take time.

Quoted for truth. They're doing everything they can to figure out just what happened. They've brought people in from the outside and they funded getting all 12 teams robots and key members of each team to FIRST HQ. <- That tends to require money, and FIRST is paying for all of it.

Just be patient.

[CalTran]

I see 2/3 of the "Eh-Team", namely my favorite blue and red box of a Canadian team...
Looks like it will be an interesting weekend. (Anybody else think the field looks oddly empty without the bridges?)

[Gregor]

I like the nice addition of the garbage can to the field. 30 points of you can balance it on a bridge?

[ratdude747]

All they need now is a large tesla coil to simulate the lightning outside :D

[Astrokid248]

...and the confetti has been removed from the field pieces.

I think we all know that this is the most important part of testing right there! :D

[RyanN]

I wish FIRST, the teams there, and everyone involved, a good luck replicating and finding the problem so it will never happen to another team again.

I'm anxious to find out what causes this problem and hope it will clear up te problems we faced at the Bayou Regional.

[Bruceb]

Hmm no updates or did I miss something?

[Al Skierkiewicz]

Bruce,
The experts are still reviewing data and filing reports. It will likely be a few weeks before the final report can be compiled. Please be patient.

[Peter Matteson]

Bruce,
The experts are still reviewing data and filing reports. It will likely be a few weeks before the final report can be compiled. Please be patient.

I don't think we are worried about seeing a final report right now. I think the general question is, were the problems replicated?

We can appreciate it will take time to go through the data and come to any real conclusions or resolution.

[Bruceb]

This from FIRST

Dear FRC Teams:



Thank you for your continued support as we conduct our examination of issues surrounding the Einstein matches from Championship.



This past weekend we duplicated, to as great an extent as possible, conditions as they were on Einstein in St. Louis. All Einstein robots and one to two representatives from each team were present. Matches were run in the same order, with teams in the same driver stations, as at Championship. In addition to 1-2 representatives from each team and FIRST staff members, highly respected independent experts and key volunteers participated in and contributed to the evaluation. Additional specialized tests were also conducted.



The FIRST staff were in awe of the Gracious Professionalism exhibited by all participants. At each turn, teams illustrated camaraderie – loaning parts, helping carry and set up others teams' robots, and even helping drive competitors’ robots when necessary.



We have learned a great deal, and have much more to learn. With the large data set to be analyzed, and a few additional tests still planned to be run, it will be several more weeks before we are prepared to draw and share our initial conclusions.



We have asked all weekend participants not to disclose details concerning weekend testing until our additional testing, results and initial conclusions are available to the entire community. The FIRST Engineering staff, with support of the external experts and the FIRST community experts are working with all deliberate speed on their analysis and recommendations. We understand everyone's desire to hear initial results and appreciate your continued patience.



Please continue to provide your experiences, thoughts, questions and concerns as we move through this process. We are checking the 2012frcfeedback@usfirst.org in-box and teams' input has already played an important role, including being the inspiration for some of the specialized testing we performed this past weekend.



Sincerely,



Jon

[Jim Wilks]

This from FIRST

We have asked all weekend participants not to disclose details concerning weekend testing until our additional testing, results and initial conclusions are available to the entire community. The FIRST Engineering staff, with support of the external experts and the FIRST community experts are working with all deliberate speed on their analysis and recommendations. We understand everyone's desire to hear initial results and appreciate your continued patience.

I always see the "not to disclose details" card played when things didn't go well. I guess there was no aha! moment when the problem showed itself again. If there was truly good news, it would be shared immediately.

Just my $.02 worth.

[IKE]

I always see the "not to disclose details" card played when things didn't go well. I guess there was no aha! moment when the problem showed itself again. If there was truly good news, it would be shared immediately.

Just my $.02 worth.

Or if you have a few likely contributors that you are trying to sort out. Or if you have a problem that you can have a solution to shortly. Or if you have a problem with no current solution and feel you need a proposed solution before you scare teams into not competing or...

There are a whole lot of reasons to not prematurely release ideas/data/anedotes. This reminds me of a neat story I read in WIRED (http://www.wired.com/techbiz/people/magazine/16-12/ff_kaminsky?currentPage=all) a few years back.

[Mike o.]

I always see the "not to disclose details" card played when things didn't go well. I guess there was no aha! moment when the problem showed itself again. If there was truly good news, it would be shared immediately.

Just my $.02 worth.

I wouldn't jump to the conclusion that there was no good news out of this past weekend. It is not an uncommon step in any engineering project/analysis to not disclose information until the analysis phase has been fully conducted. Sharing information, and especially having it go through the many phases of word of mouth can just add more confusion while they are in the concluding steps of confuting the testing and analysis. As someone who is very curious and anxious to see what has come of this investigation, I truly believe that we do need to be patient and let them fully get their work done so that we can get an accurate and concise conclusion and findings from all of this.

[Steven Donow]

I always see the "not to disclose details" card played when things didn't go well. I guess there was no aha! moment when the problem showed itself again. If there was truly good news, it would be shared immediately.

Just my $.02 worth.

Yeah, as people already said, I don't think that's an accurate conclusion to jump to. It could easily be, "We think we may have found the issue, but we're not certain that's the issue, so we don't want people to say it's the issue, then have everyone get their pitchforks out when that's not the issue" or something along those lines.

[Jon Stratis]

When we have an issue in production, my manager doesn't just want to know what happened. He wants reliable, reproducible testing AND a fix proving we identified the correct answer before he'll disseminate information to other parts of the organization. Saying you found the issue and coming back a week later and saying "just kidding" is about the worst thing an operations team can do.

[Al Skierkiewicz]

I always see the "not to disclose details" card played when things didn't go well. I guess there was no aha! moment when the problem showed itself again. If there was truly good news, it would be shared immediately.

Just my $.02 worth.

Jim,
This could not be further from the truth. I just managed to get my preliminary reports (qty. 14) filed yesterday afternoon. They were likely the easiest to compile but were needed by others to complete their work.

[Peter Matteson]

When we have an issue in production, my manager doesn't just want to know what happened. He wants reliable, reproducible testing AND a fix proving we identified the correct answer before he'll disseminate information to other parts of the organization. Saying you found the issue and coming back a week later and saying "just kidding" is about the worst thing an operations team can do.

We are the same way where I work. We review with a team of engineering fellows along the way to make sure we haven't gotten off track at each step during the DIVE process. (Define, Investigate, Verify, Ensure)

If you during your investigation that you don't verify the cause of the problem you can't ensure it won't reoccurr. I would want to make sure my data supported the conclusions before they were published, and preferably have a working solution.

[qzrrbz]

We are the same way where I work. We review with a team of engineering fellows along the way to make sure we haven't gotten off track at each step during the DIVE process. (Define, Investigate, Verify, Ensure)

If you during your investigation that you don't verify the cause of the problem you can't ensure it won't reoccurr. I would want to make sure my data supported the conclusions before they were published, and preferably have a working solution.

If you look at this as an "academic problem", you will come to a similar conclusion. It is very much looked upon badly if you publish in "the popular press" before you do so in a refereed technical journal! :)

[EricVanWyk]

Earlier this week it took me a little bit too long to realize that the reason my pen wasn't writing correctly was that it was actually a screwdriver. I'm going to have to ask for a few nights of honest sleep before signing off on a report.

[Astrokid248]

Earlier this week it took me a little bit too long to realize that the reason my pen wasn't writing correctly was that it was actually a screwdriver. I'm going to have to ask for a few nights of honest sleep before signing off on a report.

Ouch! Yeah, go catch some sleep, get a massage, have a few adult beverages. No need to kill yourself over this. Just remember that for all our whining and complaining, we actually do appreciate all your hard work and incredible effort. (That goes for anyone else present at the testing too.)

[JaneYoung]

In some ways, I think it's easier to be a NEM (non-engineering mentor) in this situation and to trust and support the process that has begun, lacking the technical expertise to fuel impatience and frustration. I have great respect for everyone involved and everyone that has to patiently wait for news and updates.

We'll get through this as a community, this we know.

Jane

[daniel_dsouza]

connection problems happen to even the most prepared...

http://www.youtube.com/watch?v=znxQOPFg2mo&feature=fvwrel

if you are in a hurry, go to 1:33.

[qzrrbz]

connection problems happen to even the most prepared...

http://www.youtube.com/watch?v=znxQOPFg2mo&feature=fvwrel

if you are in a hurry, go to 1:33.

And they figured out what was happening ... ~500 wi-fi base stations in the crowd, and the phone couldn't handle that. When they had the crowd "dim" the base stations, it worked. Hmmm...

[Jim Wilks]

It's now been over a month since the re-enactment of the Einstein saga. Not a word from FIRST since the last email stating they "needed a few weeks" to analyse data.

I maintain that if they had found something concrete they would have shared something about it.

[Mark McLeod]

Frank mentioned it again last week.

Thursday, July 5, 2012

Where’s the Einstein Report? (http://frcdirector.blogspot.com/2012/07/wheres-einstein-report.html)


Hello Teams,

I’m sure you’re wondering what the status is of the Einstein report. I can tell you that our internal staff, with a great deal of support from our outside experts and the Einstein teams themselves, have been working very hard on this. The current draft report is over 20 pages long, and includes lots of detail. Getting all these details right is taking some time, but we hope to have a final version ready for release shortly.

I did want to share with you a photo taken during Einstein Weekend, June 9-June 10th. That’s a great deal of talent represented in one place at one time, all working on a common cause!
http://4.bp.blogspot.com/-9jw1MK97qac/T_WQWbZzC-I/AAAAAAAAAI0/r7wD2DXtx0M/s320/Einstein%2BWeekend%2BPhoto.jpg (http://4.bp.blogspot.com/-9jw1MK97qac/T_WQWbZzC-I/AAAAAAAAAI0/r7wD2DXtx0M/s1600/Einstein%2BWeekend%2BPhoto.jpg)
Frank

[CalTran]

Frank mentioned it again last week.

It's good to hear that they're making progress. Probably not the progress some people are looking for, but they're not going to bust the entire wireless problem in 2 days of official testing.

[Al Skierkiewicz]

Everyone,
The report is getting very, very close. Bill's departure has added to the workload a little at HQ.

[linuxboy]

Everyone,
The report is getting very, very close. Bill's departure has added to the workload a little at HQ.

Excuse me while I go obsessively check the FRC Blog every few minutes.

[billbo911]

This is from the introduction to the PDF containing the results of the testing.

Additionally, Einstein match play suffered from an intentional act of interference.

I'm too stunned to say more!

[Alexa Stott]

Thread about the report here: http://www.chiefdelphi.com/forums/showthread.php?t=107285

[Jim Wilks]

Read the report in detail.

Lots of "might haves" and "possiblys". No real root cause whatsoever identified. The mystery lives on.

[Zach O]

Does anyone have information on what "custom electronics" 118 had an issue with?

Edit: Got the answer. I assume it's the BeagleBone.

[Alexa Stott]

Does anyone have information on what "custom electronics" 118 had an issue with?

Edit: Got the answer. I assume it's the BeagleBone.

That's all in the report. And, yes, some of it was the BeagleBone. Some of it was a bad gyro. The two worked together to create something of a perfect storm for a communication failure.

[Hjelstrom]

Read the report in detail.

Lots of "might haves" and "possiblys". No real root cause whatsoever identified. The mystery lives on.

That is definitely not the impression I got after attending the New Hampshire meeting. The FIRST and NI engineers did a fantastic job. They broke down every match for every robot and investigated every segment of time where control was lost. Every single failure was explained very well.

They are being conservative in what the claim as "intentional interference". Even in matches where a witness saw the individual doing what we now know was the hack, they described it as "likely". Only in the case where the individual *admitted* to it, is it described as "confirmed".

We left New Hampshire with the confidence that there will be many specific improvements next year and that hacks like this will be detected by the field. I honestly look forward to the improvements in the system next year.

[JaneYoung]

The report has been made public and there has been time to read and review the findings.

There was a big push for FIRST to act, and act quickly, following the troubles that arose on the 2012 Einstein field competition.

I'm interested in seeing what type of closure this has brought to this thread and to those who contributed to it. The thought, with great power comes great responsibility, has been with me for several days. I've been trying to poke holes in the thought as it applies to the work that was done by everyone committed to identifying the problems and documenting them. I can't find any holes to poke - everyone involved acted responsibly and with great integrity. The experience, and how it has been handled, provides all of us with insight towards a newly raised bar of working together, sharing knowledge and experience, and being humble and open to the process of identifying weaknesses, trouble spots, and problems. It's not an easy thing to do to keep egos in check and trust the opportunity being made available. We can all learn from these teams and these leaders who were willing to step up to that bar and, by doing so, move it to a higher level.

It's been an incredible experience to be able to follow this process from the peanut gallery. I won't put what I've experienced and learned from all of you to waste. I will use it wisely. This is a promise.

Thank you.

Jane

[Karthik]

It's been an incredible experience to be able to follow this process from the peanut gallery. I won't put what I've experienced and learned from all of you to waste. I will use it wisely. This is a promise.


I'm glad this has been an incredible experience for you. For our team, this has been one of the most heartbreaking and deflating experiences possible. I'm glad you want to revel in the sunshine here, but perhaps you should remember that there are still a lot of people who are very hurt and upset. Sure, it's just a robotics competition, but some of us actually care about the robots...

[pfreivald]

I'm glad this has been an incredible experience for you. For our team, this has been one of the most heartbreaking and deflating experiences possible. I'm glad you want to revel in the sunshine here, but perhaps you should remember that there are still a lot of people who are very hurt and upset. Sure, it's just a robotics competition, but some of us actually care about the robots...

I'm not sure I'd be able to handle the same situation with the same class you have, Karthik. The whole situation just plain sucks, and so very many of us feel for you guys.

[Taylor]

What you, and your team, experienced throughout 2012 certainly inhales audibly and I can't fathom what it's been like (I wouldn't want to). What Patrick said rings true.
I'm having trouble connecting the dots between Jane's post about learning from the engineering processes of problem solving, troubleshooting, and documentation to your taking it as an attack to your team.
Tragedies happened. We've learned from them, as individuals, as teams, as a community. Why choose to focus on the hurt and distress rather than embrace the lessons that have come? Everybody "cares about the robots" - otherwise we'd be doing science fairs. Otherwise there would have been no investigation, involving people across North America, some not even involved with the 12 Einstein teams Can't we also care about and celebrate the processes, the professionalism, the experience of healing?

[IndySam]

I'm with Taylor Karthik, I don't see how you link Jane's post with the pain your team experienced.

I can try and understand what you guys are going through but there is no way any of us can say we really can. I am truly sorry.

But we have to treat this as a learning experience and carve whatever good we can from the whole thing. I really hope you guys can too.

[Ian Curtis]

What you, and your team, experienced throughout 2012 certainly inhales audibly and I can't fathom what it's been like (I wouldn't want to). What Patrick said rings true.
I'm having trouble connecting the dots between Jane's post about learning from the engineering processes of problem solving, troubleshooting, and documentation to your taking it as an attack to your team.
Tragedies happened. We've learned from them, as individuals, as teams, as a community. Why choose to focus on the hurt and distress rather than embrace the lessons that have come? Everybody "cares about the robots" - otherwise we'd be doing science fairs. Otherwise there would have been no investigation, involving people across North America, some not even involved with the 12 Einstein teams Can't we also care about and celebrate the processes, the professionalism, the experience of healing?

Did the Eh-Team need to have their season ruined by someone trying to "make a point" about computer security for some people on the internet to learn about failure analysis, professionalism, and the experience of healing? No.

Us armchair QBs can say that we learned a lot and will put it to good use -- but did we really go through the process? I don't think we did.

[JaneYoung]

What I wrote in this thread had to do with this thread. It also had to do with the respect I've felt for those directly involved with situation. It is one that has continually shown us what we are made of as members of FRC.

Jane

[Taylor]

Did the Eh-Team need to have their season ruined by someone trying to "make a point" about computer security for some people on the internet to learn about failure analysis, professionalism, and the experience of healing? No.

Us armchair QBs can say that we learned a lot and will put it to good use -- but did we really go through the process? I don't think we did.

By 'process' I meant the shared experience of troubleshooting, finding solutions, reading and understanding the documentation that resulted. Of course we didn't go through the "process" of having our dreams shattered in a despicable and embarrassing way, and I don't believe I insinuated that any more than Jane did.
I don't consider myself an armchair QB; I consider myself a creator of tomorrows engineers and problem solvers that can learn and grow from this horrible incident.

[Billfred]

I'm glad this has been an incredible experience for you. For our team, this has been one of the most heartbreaking and deflating experiences possible. I'm glad you want to revel in the sunshine here, but perhaps you should remember that there are still a lot of people who are very hurt and upset. Sure, it's just a robotics competition, but some of us actually care about the robots...

I, too, care about the robots. As an Archimedes team, we were naturally rooting for The Eh Team. I was even more geeked when Chairman's was announced, giving the Simbots the chance to equal HOT's 6-banner 2005 season (with a GTR-W silver kicker). To see things come unglued with this intensity at this level was puzzling to our new students, shocking to our veterans.

That someone would interfere intentionally with Einstein has robbed us all of something. Those us of us watching were robbed of what could've been a spectacular finish to one of the best FRC seasons of the modern era. Twelve teams were robbed of the chance to compete on even terms. And frankly, everyone that's ever set foot on a FIRST field, from corn kernels on, was robbed of a certain innocence, the ability to say "We're better than that." Now we're forced to settle for "All but one of us are better than that, and we threw that bum out."

I hail from a state that has combined for two Einstein appearances ever. We know how rare the shot at the title is, and just how wronged everyone was this year in St. Louis. But without a time machine to hunt down that individual and slap the cell phone out of his hand, all we can do is make the most of the events that follow. This has triggered a number of procedural and technological improvements that will hopefully benefit all teams in the future. And it has served as a stern warning to anyone else thinking they'd want to interfere with a match. I agree that this came at far too high a cost, but at least we all got a sliver of something for our disappointment and heartache.

[Peter Matteson]

I've read the report and digested it at this point, and I'm posting here because I of my posts being critical of method process for the trouble shooting.

I have to say I believe they did the job correctly and had the right people there based on what I gathered from this report. I think they did the best job possible given what they had to work with.

To those of you that don't understand how it feels to be so close playing for the championship and have something outside your control end your run I really can't explain how it feels. I really feel for the entire Eh-team and understand you pain.

I had a litany of other instances I was going to post here about things that have been done wrong on Einstein over the last several years proving the point that this is not an isolated occurance, but it adds nothing to the discussion. I hope in the future FIRST pays attention to all aspects of what goes on and around the field in the future.

[Karthik]

What you, and your team, experienced throughout 2012 certainly inhales audibly and I can't fathom what it's been like (I wouldn't want to). What Patrick said rings true.
I'm having trouble connecting the dots between Jane's post about learning from the engineering processes of problem solving, troubleshooting, and documentation to your taking it as an attack to your team.
Tragedies happened. We've learned from them, as individuals, as teams, as a community. Why choose to focus on the hurt and distress rather than embrace the lessons that have come? Everybody "cares about the robots" - otherwise we'd be doing science fairs. Otherwise there would have been no investigation, involving people across North America, some not even involved with the 12 Einstein teams Can't we also care about and celebrate the processes, the professionalism, the experience of healing?

In every tragedy, there is obviously some good that comes out of the lessons learned. Please excuse me for drawing some parallels to situations which were completely beyond the scale of what we saw on Einstein, but some people in this community are so blind through their rose coloured glasses that it's the only way to make them see anything. (Again, trying to be very clear here, that I'm comparing the scope of the events on Einstein to any of these tragedies, I'm just trying to make a point painfully clear)

After Hurricane Katrina there were many lessons learned about disaster management. Did anyone think it was appropriate to say "Oh, what a beautiful day for America. I've learned so much from this!"? God, I certainly hope not.

Perhaps the peanut gallery needs to spend a little more time trying to understand what our teams are going through, and a little less time trying to put a positive spin on everything that happens. Sometimes things just suck. If I sound angry, it's because I am. I'm frankly a little tired of people trying to tell me that I should be "thankful" for Einstein. Should I be thankful that I had to wipe tears off of a student's face at a time when she should have been celebrating a Chairman's win? Was it really "beautiful" when a once in lifetime opportunity was stolen away from our team by a "hacker"? While you're all celebrating the forthcoming FMS whitepaper, perhaps you should remember that there are multiple victims here who aren't in the mood for celebration. Some members of the FIRST community, who actually get it, have been incredibly supportive, while others have just shown us a complete and utter lack of respect.

The Simbots are trying to move on, sorry if we can't do it fast as the rest of you want.

[JaneYoung]

The report has been made public and there has been time to read and review the findings.

There was a big push for FIRST to act, and act quickly, following the troubles that arose on the 2012 Einstein field competition.

I'm interested in seeing what type of closure this has brought to this thread and to those who contributed to it.


This was basically what my post was about. The rest had to do with recognizing the efforts it took on the part of FIRST, the teams involved, and those who contributed to helping the investigation move forward.

There was nothing in the post that was meant to create a false positive spin. If anything, it was meant as one of empathy and support of the process and the people impacted.

Jane

[Jay O'Donnell]

In every tragedy, there is obviously some good that comes out of the lessons learned. Please excuse me for drawing some parallels to situations which were completely beyond the scale of what we saw on Einstein, but some people in this community are so blind through their rose coloured glasses that it's the only way to make them see anything. (Again, trying to be very clear here, that I'm comparing the scope of the events on Einstein to any of these tragedies, I'm just trying to make a point painfully clear)

After Hurricane Katrina there were many lessons learned about disaster management. Did anyone think it was appropriate to say "Oh, what a beautiful day for America. I've learned so much from this!"? God, I certainly hope not.

Perhaps the peanut gallery needs to spend a little more time trying to understand what our teams are going through, and a little less time trying to put a positive spin on everything that happens. Sometimes things just suck. If I sound angry, it's because I am. I'm frankly a little tired of people trying to tell me that I should be "thankful" for Einstein. Should I be thankful that I had to wipe tears off of a student's face at a time when she should have been celebrating a Chairman's win? Was it really "beautiful" when a once in lifetime opportunity was stolen away from our team by a "hacker"? While you're all celebrating the forthcoming FMS whitepaper, perhaps you should remember that there are multiple victims here who aren't in the mood for celebration. Some members of the FIRST community, who actually get it, have been incredibly supportive, while others have just shown us a complete and utter lack of respect.

The Simbots are trying to move on, sorry if we can't do it fast as the rest of you want.


Thank you for saying this, I think it really puts the enormity of this tragedy in perspective. FIRST is a huge part of mylife, and I'm sure it is the same for most kids on your team. I especially feel for the seniors who will never have another chance at this as a student, and I hope for your teams sake you make a quick return to Einstein.

Simbotics is one of best teams in FIRST, both on and off the field. What happened is extremely unfortunate for you guys. I completely agree that this is not something that you will be able to get over quickly, and the FIRST community is here for you. As we all know, time heals all wounds, but for right now, you every reason to be upset. Your team, along with the other 11 on Einstein, have been the most professional group I have ever seen, I know I wouldn't be able to handle it as well as you all are.

[Racer26]

For those having trouble grasping the enormity of Karthik's posts, I'm gonna put them in perspective for you.

2012 Simbotics Chairman's Video (http://www.simbotics.org/media/videos/2012/chairmans/2012-chairmans-video-i-remember)

Watch it. Again. Of all the teams in FIRST, including past HoF teams, I can't think of a team that "gets it" any better than 1114. The video says it all. 1114 is a CLASS ACT.

Karthik and the rest of 1114 have pretty much dedicated their lives to this program. There is a reason their CCA this year was LONG overdue.

To have their chance at taking home the double crown stolen away by some fool with a cellphone is devastating. I don't know strong enough words to describe the emotions I felt, never mind what the members of 1114, 2056, 4334, and the rest of the Einstein teams have felt.

Imagine how 4334 feels. They're rookies! They "get it" too. Won Rookie-All-Star at GTREast to earn their seat. They're trying to get a western Canada regional started NEXT YEAR. For those playing the home game, there are currently TWO teams in western Canada, 1482, and them. Its an absolute tragedy that their stellar first FIRST season came to an end like this. I have to imagine its making them question all the time and money they've put in.

[CLandrum3081]

I am shocked and saddened by what happened. I think we all feel a little betrayed by this. As one of the teams in Archimedes, what we did was cheer on 1114 and the Archimedes champs because that's what FIRST is about. As a first year, I was excited about Coopertition and everything this organization had to offer. To have that be undermined by a hacker... Who would stoop so low as to mess with an alliance that had a rookie team and a potential double crown winner?! Or even to mess with the other teams?! Communication problems happen, but now paranoia will set in and every time there is one, teams will jump to conclusions because someone wanted to mess with a championship that was supposed to be the most exciting weekend of many of our lives. It'll be interesting to see how 2013 regional competitions deal with this issue - will teams still trust each other, or will they suspect that anyone with a computer or an iPad or a cell phone is a hacker? Will successful teams, who thought they'd been supported by the other teams, now distrust them because they think others are out to get them? I think anyone with half a heart is shocked and disappointed by what occurred. I remember watching Einstein knowing something was wrong, but out of my own trust in everyone there, not even letting the thought of intentional interference cross my mind. My heart truly goes out to those affected, especially 4334, 2056, and the Simbots of 1114. Hopefully future years will not be plagued by this issue and this wound can heal and we can move on, but many of the students hurt by this will be FIRST alumni before this happens, if it does.

[JaneYoung]

This thought is only partially formed and I may come back in and edit a million times until I have it the way I'd like it to be but I'm going to give it a try now.

We can find many discussions here in CD regarding taking risks and facing challenges. We can find discussions about how those challenges are brought about by the continual advancement of technology and how it impacts our world. In this case, our world is FRC. The risks and challenges are always there but sometimes - the discussions are theoretical. Sometimes, the risks are small. Sometimes, the challenges are worked through and the teams achieve a goal.

Sometimes, the risks and the challenges become very real. Technology shows us what it can and cannot do. I think that we are looking at that right now. There is a reason that FRC has core values. There is a reason that Dr. Woodie Flowers has spent a good deal of his professional life talking about risks, challenges, ethics, core values, Gracious Professionalism, and kindness. None of those have changed. They have been challenged and we are risking losing sight of them and, in doing so - ignoring their strengths. This is as dangerous as the misuse of technology and it is creating some of the stresses and concerns that we are all feeling here in CD.

There is a reason for the Woodie Flowers Award. It is a very serious reason. There are many mentors in many FRC teams who hold the values, knowledge, and respect for core values - that qualify them to be outstanding nominees for the Woodie Flowers Award, capable of joining with our WFFAs and WFAs and their commitment, influence, and impact. This is very important.

The news is fresh and has sent a shockwave through the CD community and members of FIRST. When the time of recovery begins then a time of rebuilding will begin. Trust will be a part of the rebuilding period and I believe that FIRST is working very hard now to help all of us when we are ready to help to rebuild the trust. We can certainly undermine with worries and doubt and criticism. We can also work within our teams to deepen the respect and understanding of FRC's core values and what they mean to the team and to the community and how, when a bad situation occurs that threatens our security - we have to lean on those core values and share them with each other.

We have members of our FIRST family who are suffering terribly. They are saddened, angry, betrayed, and feeling great loss. But - those members will work through it in their own time, leaning on the core values and on their community. Their community is ... us. All of us. We'll get through this.

Jane

[Siri]

If I sound angry, it's because I am. I'm frankly a little tired of people trying to tell me that I should be "thankful" for Einstein. Should I be thankful that I had to wipe tears off of a student's face at a time when she should have been celebrating a Chairman's win? Was it really "beautiful" when a once in lifetime opportunity was stolen away from our team by a "hacker"?I just had to say that this line is absolutely heart-wrenching. Your team, 2056, and 4334 and the Einstein teams, are such class acts. I know I can't claim to fully comprehend your pain despite my best efforts, but your sentiments are truly powerful. No one should dictate that speed at which the Eh-Team 'moves on', and being thankful for FIRST's changes and to all those that made them possible should not morph into a desire to be thankful or accepting for the tragedy that caused it.