FIRST's statement on Einstein

[rachelholladay]

We had one camera connected at the Bayou Regional and never had comm issues. We added a second camera for CMP and connected for our practice match perfectly. We went on for match #1 and couldnt connect. (I must say as an ex-FTA assistant, hearing them say "Bypass 1912" was one of the worst feelings ever). After that match I switched out our radio (all of our radios were reprogrammed for CMP) and went to match #2. We connect for approximately ten second and then it dropped. The FTA said "I have no confidence in your radio" and we were bypassed again. I then received very generous help from some FTAs and control experts (in another thread I explain their extreme kindness) who ended up downgrading my radio firmware version from 1.4 to 1.2 and then we decided to disconnect our two radios. By the odd event of a balsa wood airplane coming on field we were able to replay match #1, but this time with our dual camera system disconnected entirely. From then on I never had issues connecting to the field and never replugged in either camera. The FTAs mentioned that dual camera had caused problems for other teams.

I dont know if the dual camera issue was the problem for anyone else. All i know is, my controls freshmen have a sworn vengeance against camera now..

[JB987]

Originally Posted by rachelholladay

We had one camera connected at the Bayou Regional and never had comm issues. We added a second camera for CMP and connected for our practice match perfectly. We went on for match #1 and couldnt connect. (I must say as an ex-FTA assistant, hearing them say "Bypass 1912" was one of the worst feelings ever). After that match I switched out our radio (all of our radios were reprogrammed for CMP) and went to match #2. We connect for approximately ten second and then it dropped. The FTA said "I have no confidence in your radio" and we were bypassed again. I then received very generous help from some FTAs and control experts (in another thread I explain their extreme kindness) who ended up downgrading my radio firmware version from 1.4 to 1.2 and then we decided to disconnect our two radios. By the odd event of a balsa wood airplane coming on field we were able to replay match #1, but this time with our dual camera system disconnected entirely. From then on I never had issues connecting to the field and never replugged in either camera. The FTAs mentioned that dual camera had caused problems for other teams.

I dont know if the dual camera issue was the problem for anyone else. All i know is, my controls freshmen have a sworn vengeance against camera now..

987 ran the entire season (65+/- matches including practice matches) with 2 cameras and a Kinect without any coms dropped until our matches on Einstein...

[Citrus Dad]

Originally Posted by Grim Tuesday

I wonder why these issues only surface in eliminations? I can't remember another time 1717 wasn't working. Makes me think back to Leeland's comments in other threads, proposing something to do with the increased network bandwidth of highly capable teams (image processing, cameras, etc...)

1717 went dead in a qualifying match in which 1662 saved their gravy by scoring a zillion baskets.

An interesting question is whether the combination of so many 'bots using so much band width was higher than in a typical regional.

[Citrus Dad]

Originally Posted by Natchez

Please allow me to respectfully disagree. The Jones Dome has a disaster plan in place that includes how to handle a nearby tornado and, although I can not find the plan online anywhere, I suspect that the plan calls for partial shelter-in-place and partial evacuation to the tunnels when practical. The plan should not, and probably does not, call for withholding information from the people who will be directly affected.

In my view, what should have been done is that the field level should have been evacuated to the tunnel area leaving the floor bear and the people in the stands should have prepared for the roof to blow off and the glass in the exterior to shatter .... I don't really know if that is to shelter-in-place or proceed orderly to the tunnels; that is why the disaster plan is in place. FIRST should have followed the Jones Dome Disaster Plan.

Consider this, if you were at a Rams game and it sounded like a train was about to hit the stadium, do you really believe that it is right to let the players, staffs, & others remain on the field and hope bad things don't happen when there is very good shelter only a few hundred feet away. Also, if we enacted a "hush hush" policy in these circumstances, it seems unfair that we would not warn the people who decided to go to the bathroom or get a hotdog in a highly glassed area.

The roof blowing off a dome is a real fear and when a couple of panels get ripped off, things escalate in a hurry. In a dome, being under the primary roof is the first or second most dangerous place to be located.

Now, please allow me to defend my actions. I was on the floor and heard the sound of a train about to run into the Jones Dome from all directions. Knowing that bad things were happening outside and the fact that I did not have my handy crystal ball with me, I took action to remove participants and patrons from underneath the primary roof and into an "enclosed" area. First, I identified where people could go and then encouraged them to move to the tunnels; okay, I did not encourage the Robonauts but demanded the Robonauts get a move on it. I texted all of the Robonauts mentors (we had a group set up so it only took me a few seconds to text them) and asked them to get everyone to shelter. FIRST, obviously having their crystal ball in hand, told people that everything would be okay ..... just trust us. In review, I suspect my actions were more closely in line with the Jones Dome Disaster Plan for tornadoes than FIRST's actions.

When I read things like "the LAST thing FIRST should have done was make an announcement" because we can't trust people, I wonder what America is coming to,
Lucien

We were up high in the stands. The loud noise we heard was hail, not high winds or tornado, nor thunder. The situation didn't seem quite that dire. Also, the event at Busch appears to be high winds, not a tornado.

Regardless, probably taking more appropriate actions would have been better.

[Chickenonastick]

Originally Posted by rachelholladay

We had one camera connected at the Bayou Regional and never had comm issues. We added a second camera for CMP and connected for our practice match perfectly. We went on for match #1 and couldnt connect. (I must say as an ex-FTA assistant, hearing them say "Bypass 1912" was one of the worst feelings ever). After that match I switched out our radio (all of our radios were reprogrammed for CMP) and went to match #2. We connect for approximately ten second and then it dropped. The FTA said "I have no confidence in your radio" and we were bypassed again. I then received very generous help from some FTAs and control experts (in another thread I explain their extreme kindness) who ended up downgrading my radio firmware version from 1.4 to 1.2 and then we decided to disconnect our two radios. By the odd event of a balsa wood airplane coming on field we were able to replay match #1, but this time with our dual camera system disconnected entirely. From then on I never had issues connecting to the field and never replugged in either camera. The FTAs mentioned that dual camera had caused problems for other teams.

I dont know if the dual camera issue was the problem for anyone else. All i know is, my controls freshmen have a sworn vengeance against camera now..

We had a similar issue at Davis in which our comms were dropping after a few seconds into the match. It turned out that absolutely nothing was wrong with our radio -- the resolution of our camera feed, when using dual cameras, was simply too high, causing the memory on either the cRIO or driver station computer (I'll get back to you on which one) to overload, which resulted in packet loss and a lost connection.

[billbo911]

Originally Posted by Chickenonastick

We had a similar issue at Davis in which our comms were dropping after a few seconds into the match. It turned out that absolutely nothing was wrong with our radio -- the resolution of our camera feed, when using dual cameras, was simply too high, causing the memory on either the cRIO or driver station computer (I'll get back to you on which one) to overload, which resulted in packet loss and a lost connection.

Along these lines....... The only time in ~30 matches this year we were dead on the field, was NOT because of comms loss. We had errors that continued to accumulate on the DS. Now I am not certain the exact mechanism that caused us to sit dead on the field, but we suspect that we had filled up a buffer, HD space, or.... Whatever it was, as soon as we cleared the error queue, or robot was able to move again.

Once we figured this out, clearing the queue became part of our start up routine for every match.

After this, we also found the root cause of 2 out of the three errors that kept happening and corrected it. That reduced the number of errors being logged.

Whether this is related to what other teams were experiencing or not, I don't know. I put it out there to add to the clues that might help get to the bottom of the "comm errors".

[Bryscus]

Can the teams on Einstein post what version of firmware was on their bridge? I know that at the South Florida Regional (our second event - which coincidentally used the same field as our first event, the Orlando Regional), they downgraded everyone to 1.21, so that's what we were using. I'm wondering if anyone was running 1.4?

Also, after talking with some of the tech gurus and some other teams at SFR, it was recommended to us that we log off or reboot the laptop after every match as some of the network data was not being released. Did anyone else do this?

SPAM had a camera on board that only streamed to the DS running SmartDashboard and something like 8-12 pieces of data on the screen. We originally were tracking with a second camera on the cRIO, but the driver was better than our turning algorithm so we scrapped it at the end of the SFR.

- Bryce

[Holtzman]

Originally Posted by Bryscus

Can the teams on Einstein post what version of firmware was on their bridge?
- Bryce

1114 and 2056 both had 1.21

[Undertones]

I have a lengthy post regarding this topic already, but there's a couple things I still want to say.

The champions are champions now, and will remain so. Might as well make the best out of that situation. Team 4334 heartily congratulates the champions.

However, this was a world championship competition. The people who made the calls they did shifted the immediate blame from the FIRST to the teams in the eyes of the average spectator.

Imagine if at the Stanley Cup finals, there was a massive hole in the ice, that at times, detained half the players on the ice and kept them from moving at all. I sort of think they might stop the gameplay and fix the ice.

While everyone's gracious professionalism remained remarkably high on the floor considering what happened, it doesn't take away from this that it is completely unacceptable. I mean, if you want to break it down, the teams played a little under 20 matches at the championships, at a $5000 entry fee, we all paid $250 to get screwed by the field. I've already said that I'm not going to play the "well this would have happened if Einstein wasn't broken" game, because, well it's really not very gracious, but nonetheless I feel cheated out of an opportunity to show the world what we could do. The Maple Leaf Alliance had an amazing triple balance, and although I am biased because I was part of that alliance, I think we can all agree it was impressive nonetheless.

If you're interested, you can see my full post on what happened on Einstein here.
http://www.chiefdelphi.com/forums/sh...&postcount=182

Mac

[frdrake]

Originally Posted by Undertones

I have a lengthy post regarding this topic already, but there's a couple things I still want to say.

The champions are champions now, and will remain so. Might as well make the best out of that situation. Team 4334 heartily congratulates the champions.

However, this was a world championship competition. The people who made the calls they did shifted the immediate blame from the FIRST to the teams in the eyes of the average spectator.

Imagine if at the Stanley Cup finals, there was a massive hole in the ice, that at times, detained half the players on the ice and kept them from moving at all. I sort of think they might stop the gameplay and fix the ice.

While everyone's gracious professionalism remained remarkably high on the floor considering what happened, it doesn't take away from this that it is completely unacceptable. I mean, if you want to break it down, the teams played a little under 20 matches at the championships, at a $5000 entry fee, we all paid $250 to get screwed by the field. I've already said that I'm not going to play the "well this would have happened if Einstein wasn't broken" game, because, well it's really not very gracious, but nonetheless I feel cheated out of an opportunity to show the world what we could do. The Maple Leaf Alliance had an amazing triple balance, and although I am biased because I was part of that alliance, I think we can all agree it was impressive nonetheless.

If you're interested, you can see my full post on what happened on Einstein here.
http://www.chiefdelphi.com/forums/sh...&postcount=182

Mac

Lots of us non Canadians don't get hockey references

[rachelholladay]

Originally Posted by Chickenonastick

We had a similar issue at Davis in which our comms were dropping after a few seconds into the match. It turned out that absolutely nothing was wrong with our radio -- the resolution of our camera feed, when using dual cameras, was simply too high, causing the memory on either the cRIO or driver station computer (I'll get back to you on which one) to overload, which resulted in packet loss and a lost connection.

We trying to find the root of our error, we looked at our resolution. I believe we had both cameras at the default of 30. Based on the suggestion of an FTA both were scaled down to 5. However, we kept the cameras unplugged and therefore never tested if if was the resolution that gave us problems. I could have, but I didn't want to risk being dead in a match.

[lemiant]

Over-all we've had an awesome season and I've documented much of its awesome-ness here: http://www.chiefdelphi.com/forums/sh...hreadid=106240 . However I have many issues with how Einstein was handled, and after giving myself a bit of time to catch my breath I thought I would add my thoughts to the debate:

Let’s start off this way. While I love FIRST and I know that it is an incredible organization, and while I will continue to support it in the future... I believe that they messed up, REALLY bad! Almost every aspect of how Einstein was handled was wrong in my mind and FIRST has some serious fixing to do. We can look at all sorts of impacts, but the one I am most familiar with is what it did to our team. We were the first rookie team to make Einstein since 2009, but the night after that monumental accomplishment there was not a smile to be found. We are starting to get over it, but for the first few days, that screw up overshadowed everything else about the championship in the place it matters most... the hearts of the students.

The mistakes started at the beginning of the season and many people have talked about them already: FIRST really should have stopped considering themselves infallible and implemented proper troubleshooting as soon as the problems surfaced; They also should have had more contingencies to deal with the problem.

Their next step was the one thing I applaud them for. When the first two matches on Einstein went bad, FIRST did the right thing and replayed them (I was really surprised by the amount of cheering when they announced this). However the thing that gets me is that, to the best of our knowledge, nothing was actually done to fix the field between that and the next match. What FIRST actually did to try and fix the problem is something we can only speculate about, because there was, and still is, very little transparency about what happened.

Once it was readily apparent that the next set of matched had gone no better, FIRST decided that they would power through anyways and crown a champion regardless of the fact that all of the matches were more heavily influenced by the field than any other single factor. At this point FIRST should have called it. Our entire alliance was standing there incredulous when they announced that the scores from SF2-1 would stand. I believe that if they cannot provide a field that is fair for all of the teams and lives up to the specs they provided, then FIRST should have stopped playing matches until they could, in this case that would have meant crowning all 4 alliances world champions. It’s rather bold of me to suggest that we should have been given that honor, and I think our team already had so much good luck going for us that we really have nothing to complain about. However it is absolutely ridiculous for FIRST to suggest that our two incredible alliance partners, 1114/2056, deserved to lose those matches in the way they did. Since there is no validity to the matches that went down on Einstein I really wonder why FIRST insisted on holding them at all.

Regardless I would like to extend my congratulations to 25, 16, 180, 233, 987, 207, 118, 548 and 2194 who all built incredible machines and worked extremely hard to get to Einstein. Despite my opinion, given what actually happened, I recognize that there is no way to go back and make everyone world champions, as much as I wish there was.

Throughout it was surprising and insulting that FIRST kept pretending there wasn’t a problem. Our driver already addressed this in another post so I won’t harp on it http://www.chiefdelphi.com/forums/sh...&postcount=182

I may also be in the minority of people who are not content with FIRST’s apology. In my mind given such a monumental failure I would expect something more than a statement which basically boils down to “sorry for the inconvenience”. Maybe:

“We recognize that we screwed up big time. Our apologies to everyone at the championship event and to the Einstein teams in particular for what happened on the Einstein field. Our FMS failed in a way we did not expect and significantly lowered the quality of play as well as affecting the outcome of the championship event. We deeply regret what happened and will not allow it to happen again”

Followed by opening a website to detail the results of their investigation into the field failure, which was immediately populated with the error logs from the Einstein field along with full information about the inner workings of the field electronics. In addition there would have to be pre-set deadlines for further reporting. To deal with the damage that has already been done, I'm a big fan of giving all the Einstein teams qualification and free registration, but I’m rather too biased for that to hold much weight. Something to remember though, is that collectively the 12 of us paid $60,000 dollars to attend the event (nevernmind the registration of everyone watching) and got a completely ganked product. If this was the real world, FIRST would be giving a full refund and doing some serious damage control (remember Toyota?).

FIRST has a massive black eye and the only way to move past it is admitting it and making serious changes. Something they haven’t done so far, but which I still hope will happen soon.

That’s a lot of text, but it's all part of a season which was awesome for us personally! But in which, the organization itself screwed up bad. I’m still fully behind their mission, and if they get their act together, as I have faith such an intelligent group of individuals will, then I forgive everything and fully support their organization as well.

Here’s to an even better 2013!

- Alex

[Meredith Novak]

Originally Posted by Bryscus

Can the teams on Einstein post what version of firmware was on their bridge? I know that at the South Florida Regional (our second event - which coincidentally used the same field as our first event, the Orlando Regional), they downgraded everyone to 1.21, so that's what we were using. I'm wondering if anyone was running 1.4?

We were using a new bridge, rev. B1, with 2.0 firmware. We never died throughout the entire event. JT and I wired a bridge backward and burned it up, so we ordered the new one. Maybe it was a fortunate mistake...

[Undertones]

Originally Posted by lemiant

...Our entire alliance was standing there incredulous when they announced that the scores from SF2-1 would stand.

Thank you Alex, you found the word I've been looking to describe my reaction to all of this.

Incredulous. What a great word, and how descriptive it is of this situation and my reaction to it.

You know the stereotypical teen reaction to a tough breakup? That's pretty much how I've handled this (minus the crying and threats).

Anyways, everyone already knows my opinions so I'll stop getting in on posts.

Mac

[mjgard]

These comm issues could have almost been predicted. In the early matches on Curie we got so far behind Thursday because of issues and they finally got it working smoothly. Trying to run only a few matches on Einstein without working out any issues by having practice matches on the field seems to be asking for problems. Another solution would of been to leave the Curie or Gallaleo field assembled until the end of the competition incase these comm issues occured they could have moved to a field that was not having the same problems earlier in the day and see if that would have solved the problems. Those fields were 90% apart by the end of the Championship when they could of been used as a back up once a problem was seen. A lot of possible issues could of been the problem, including the storm outside, but having another field as a back up might have been at least worth a try.

Something else to consider. With the large number of people and cell phones in the stands during the championships was there interference. I am sure many people had mobile hot spots running in the stands and others at least were using data. Could all of these signals caused a problem with the field?