Einstein Field issues Handled correctly?

[Teamcodeorange]

Inferring nothing about the teams that played in the finals I believe the field problems were handled very poorly. I know there are time concerns but that was unacceptable. What do you think?


Edit: btw this happened to us (Code Orange 3476) twice on the Archimedes field.

[ratdude747]

Agreed. They should have moved to another field and tried again.

[Andrew Lawrence]

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

[ratdude747]

All other fields were torn down,

That is also a problem... they need to keep at least one other field ready to go in case this kind of thing happens...

[Racer26]

They should have gone to Galileo after the first replay when 118 sat dead in the replayed SF1.1

[Steven Donow]

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

The other fields were not torn down, but even with that in mind, there really was nothing that they (logistically) could do. Add that to the huge storm, they handled it the only way they could. They couldn't move everyone, it would be a logistical mess. They couldn't change the field or change the red alliance station, that would take way too long. The only thing that I think they could have possibly done(and even this is a longshot, I think the FMS would prevent it from going smoothly) would be having the red and blue alliances switch so that they could make ONE HUNDRED PERCENT sure that it was the field, and not the robots*



*I'm not saying it was the robots, but this would make it so that there's absolutely no way they could say, 'It's you, not the field'

[Robotmmm]

I feel bad for 180, 25, and 16. Sure they were declared the winners, but it will forever be a tainted win. We will never know who the real winners should have been.

[Tetraman]

That is also a problem... they need to keep at least one other field ready to go in case this kind of thing happens...

I'm sure for next year, this is going to be on the list of "things to make happen."

For me, I can't be sure if they did the right thing or not. Under the circomstances, they probably did while in retrospect they didn't. Obviously there wasn't some backup plan in the event of the Einstein field having such errors, if it was the field at all. It might have been a "Einstein, the unsinkable field" mentality that it's been given time and time again, so having or needing a backup plan in case all goes haywire wasn't on anyone's mind.

That being said, as a volunteer myself I tend to give credit to the other volunteers and give them the benefit of the doubt that they did all they could. I won't say the field was to blame, but I would site the field as a probable culprit.

TL;DR, I don't know. But at least we know next year will have a lot more backup plans just in case of similar situations in the future, as well as a series of steps that would be taken to handle the issues correctly.

[ErikEdhlund]

Based on the situation they should have taken a break (30-45 minutes) to correctly review the issue and to allow the teams to test their connections. Just allow the teams spectating and competing to go break for a dinner break instead of just allowing the teams to suffer. How they handled the situation was very unprofessional as they did not even acknowledge after the replay any remaining issues and just remained quiet for an extended period of time before announcing the results.

[nlknauss]

I feel bad for 180, 25, and 16. Sure they were declared the winners, but it will forever be a tainted win. We will never know who the real winners should have been.

I don't know that I would call it a tainted win because we don't have any real information (other than our observations) to support that it is a field problem. Yes, this is the first time we've seen this kind of failure on Einstein but I don't think anything should be taken away from the winning alliance. We'll probably learn more soon...hopefully.

Did they handle it right? I'd say that they handled it well after the first 2 failures by replaying the matches. After that, I'm sure they did the best they could. No one wants to see robot bricks on any field at any level.

[George Nishimura]

Logistically, they did almost everything they could have. Replaying the first two semis was the only thing they could, I doubt from that point on they could realistically have replayed more matches, or used more time. They have a schedule to keep to.

However I do agree that they should have swapped sides, I've never been a volunteer but I imagine that's not too burdensome to change, and they should come up with a back-up plan in case of failure.

All in all it was the problem itself that was disappointing, not necessarily the solution. How they handle this problem hasn't been concluded, so I'd await further judgement. If they come out with some sort of explanation, it would help immensely.

[plnyyanks]

I may take some flack for this, so I'll try and explain my thought process as thoroughly as possible. I'm not yet entirely convinced that it's the field/FIRST's problem, and I don't think I know enough to make an educated call. I agree that it looks very suspicious, but this is how I see it.

These problems are very hard to diagnose the root cause of. There have been numerous long threads here listing as much evidence as we can, and many of the greatest minds in FRC haven't yet figured it out. These are not easy problems to solve, and I have full confidence that the FRC staff is working as hard as they can to work this out.

Many times, it is actually a robot problem. There have been loose connectors, USB issues, and more. After the CT Regional (which had its fair share of communication issues), I spent some time talking to the FTA. We agreed that some (but not all) of the problems were team caused - IIRC, there was a team that admitted its own fault on this. For the rest of the problems, the FTA said that he simply didn't know. There are so many variables constantly changing, and it's extremely hard to debug with the tools available. WiFi networks can be very tricky, especially regarding interference. I'm not saying it's FIRST's problem, and I'm not saying it's the teams' problem. Just trying to look at the big picture here.

Now, with a little more respect to today's Einstein matches, specifically. Yes, it is fishy that for some teams, this was the first sign of communication trouble. Yes, it is fishy that the robot at Red 2 was totally dead for 3+ matches in a row. Yes, this is a huge problem - just look at Dean and Woodie's facial expressions during their speeches, they both looked REALLY stressed out, and with good reason.

But if you were in their situation, what would you have done? Consider the weather: electrical storms and hail. That can't be good for the field networks. Now, I'm not in St. Louis now, but I think the inclement weather started right around the same time as the Einstein matches (EDIT: apparently, it started for the finals (http://www.chiefdelphi.com/forums/showpost.php?p=1163544&postcount=5http://www.chiefdelphi.com/forums/showpost.php?p=1163544&postcount=5). But it's still possible for there to be electromagnetic interference before and after the hail started, since the storm was still in the area). Thunderstorms and field trouble are correlated, in this case (but this doesn't imply causation (http://xkcd.com/552/), however). But I think the weather played a large part in the trouble - the field network had to have had some problems with all the electrical interference in the air, from both the weather and people's devices in the stands. I would love to see some data from the FMS logs, DS logs from teams competing on Einstein, and any data about wifi traffic in the area. It may shed some light on the problems.

I do think they should have considered moving to another field or switching red/blue, and I hope they did consider that. But they decided not to, for whatever reason, and I remain confident that they chose this with a rational explanation; I'll give them the benefit of the doubt on that one.

Let's be frank: getting eliminated due to connection issues is just an absolutely awful way to go. There's no ifs, ands, or buts about it. It sucks. A lot. We see that we have a problem. Complaining won't solve anything. Let's look at the big picture, and try to see how all the many variables fit into place. The problem is complicated enough on its own. Let's put our heads together, collect as much hard data points as we can, and try to fix things.

tl;dr: I'm not going to pass judgement on how things were handled and whose fault it was. I wasn't in St. Louis, and I don't know all the facts that the field people knew. Until I know as much as possible, I'm going to refrain from saying it was either party's fault. Let's try to get as much data as possible into the open and work together to solve the problem instead of mudslinging on whose fault it was.

EDIT x2: (http://www.chiefdelphi.com/forums/showpost.php?p=1163555&postcount=33) I'm not saying that it's nobody's fault. It's possible that it is entirely FIRST's fault. But I don't feel comfortable making that call with the current information available to me.

[Sean Raia]

So let me get this right... The general consensus is that it was the Einstein field in particular causing the issue?

Because the way I see it, it was more likely caused by the large number of 3g phone signals creating massive wireless noise.

[efoote868]

180, 25, and 16 played and won on the red alliance. The field issues seemed to be just with the red alliance, so I would not consider it a "tainted" win.

I cheered when they announced replaying both matches. That was the correct decision, and as far as I know, the only decision they could have made.

[Kevin Sevcik]

All in all it was the problem itself that was disappointing, not necessarily the solution. How they handle this problem hasn't been concluded, so I'd await further judgement. If they come out with some sort of explanation, it would help immensely.This.

Yes, the issues on Einstein were horrible, but after the first replay all you could really do is watch the train crash happen. The single most important thing is how FIRST addresses the problem in the coming weeks and months. If there isn't any comment or statement out of FIRST within a week or two, then I think we should all be highly concerned.

[Sean Raia]

180, 25, and 16 played and won on the red alliance. The field issues seemed to be just with the red alliance, so I would not consider it a "tainted" win.

I cheered when they announced replaying both matches. That was the correct decision, and as far as I know, the only decision they could have made.

Simply not true. Blue alliance had com issues as well. Although strangely they surfaced later in the tournament.

[plnyyanks]

Because the way I see it, it was more likely caused by the large number of 3g phone signals creating massive wireless noise.

That and the weather...
Consider the weather: electrical storms and hail. That can't be good for the field networks.

[ErikEdhlund]

Now, I'm not in St. Louis now, but I think the inclement weather started right around the same time as the Einstein matches. Thunderstorms and field trouble are correlated, in this case (but this doesn't imply causation (http://xkcd.com/552/), however). But I think the weather played a large part in the trouble - the field network had to have had some problems with all the electrical interference in the air, from both the weather and people's devices in the stands.

I am curious as to how it would be possible for electrical interference in an enclosed dome where there was no apparent power surges or outages. Not to sound rude but is that even possible?

[Sean Raia]

That and the weather...

This makes a lot of sense. To me, a field doesnt simply "go bad" for random driver stations at random times. It must have been an external cause

[Walter Deitzler]

I am curious as to how it would be possible for electrical interference in an enclosed dome where there was no apparent power surges or outages. Not to sound rude but is that even possible?

I don't know if its possible or not, but after living in ST. Louis my whole life, I have seen this stuff happen during bad weather. Sometimes during bad weather I will lose wireless, without a power surge or outage. I wouldn't put it past the Midwest...

[Sean Raia]

http://cdn.memegenerator.net/instances/400x/19625152.jpg

[efoote868]

If only we could go back to IFI's radio system... wifi problems solved.

[plnyyanks]

I am curious as to how it would be possible for electrical interference in an enclosed dome where there was no apparent power surges or outages. Not to sound rude but is that even possible?

This is how I see it. Feel free to correct me if I'm wrong. I'm not an expert on this stuff, but I don't think many of us here are. I'll do the best I can. Wifi networks use electromagnetic waves to transmit data (yes, oversimplified, I know). Thunderstorms emit lots of electromagnetic radiation (https://en.wikipedia.org/wiki/Atmospheric_noise#Lightning). This interference (https://en.wikipedia.org/wiki/Electromagnetic_interference), I think, had the possibility to cause some serious problems.

The dome is not impervious to electromagnetic radiation. Some interference can still get through (albeit less, compared to open space). Think about how, say, cell phones cause wireless interference: more electromagnetic radiation in the air, interfering with your network. This doesn't cause any power outages or surges, but still can take down the field network. It's similar with regard to that kind of atmospheric noise - it doesn't have to be enough for a power surge to impact the network.

[nlknauss]

That and the weather...

If this possible, maybe we'll see weather delays in FRC?! Like in sports, they could show highlights or replays of past games for the audience as they wait. :D

[BrendanB]

What gets me is that

1. Einstein was never used. Do they ever bring out 6 robots to check a practice match or do they hope they assembled it right?

2. The robots were all in working order before coming to Einstein.

3. Some robots worked in some matches then died later others not at all.

FIRST couldn't find a problem, but there is a problem and when you are down to the final four you can't just decide to keep playing through matches when there is a problem like this.

What I would have suggested (please note I will admit I'm not an expert at the FMS or electrical components in general) is that they run a test match in between awards that bypasses the FMS system. Just like a team would if they wanted to run a practice match at home. Each head ref enables the robots in practice mode so they run autonomous followed by teleop. Allow the teams to drive around the field.

Everybody works fine and 118 moves then you have field error if someone dies and 118 is still not moving and robots loose comms then you have robot failure. If the field is at error I've always wondered why they could just run a match as listed above where the refs start each robot at the exact same time. Understandably you might have some robots running a tenth of a second longer than others but I see that as better than not at all.

Again the above is a real long shot but it is an option other than running matches and hoping everyone is okay because you can't find a problem. I was at the CT regional where 118 and other had problems and they traced them to USB and robot errors not the field and I didn't see that here unless they kept it quiet.

Overall very sad to see robots perform amazing in their divisions and not agree with that field.

[Chi Meson]

I would be curious to know where the Einstein field was during regionals. I'm treading very lightly here, because we had just come to terms with our season and accepted fate for what it is: a slippery eel. For those who just watched 118, you saw exactly what happened in the CT semis. Watching the Robonauts brick again, and again, was terribly sad. They have such a beautiful robot, and tripling with them was always picture perfect.

The field in CT was the same as the one in NYC , which also had persistent comm issues, predominantly on the red side. So, where has Einstein been?

[BrendanB]

I would be curious to know where the Einstein field was during regionals. I'm treading very lightly here, because we had just come to terms with our season and accepted fate for what it is: a slippery eel. For those who just watched 188, you saw exactly what happened in the CT semis. Watching the Robonauts brick again, and again, was terribly sad. They have such a beautiful robot, and tripling with them was always picture perfect.

The field in CT was the same as the one in NYC , which also had persistent comm issues, predominantly on the red side. So, where has Einstein been?

Einstein has been the backup field sitting in a warehouse all season, AKA never been used. Part of me believes that the robots we saw were the first robots ever to connect to that FMS. I don't know much about the system but I don't believe a FIRST field is working properly and is ready for competition until I see 6 robots play a match with no issues.

[ErikEdhlund]

This is how I see it. Feel free to correct me if I'm wrong. I'm not an expert on this stuff, but I don't think many of us here are. I'll do the best I can. Wifi networks use electromagnetic waves to transmit data (yes, oversimplified, I know). Thunderstorms emit lots of electromagnetic radiation (https://en.wikipedia.org/wiki/Atmospheric_noise#Lightning). This interference (https://en.wikipedia.org/wiki/Electromagnetic_interference), I think, had the possibility to cause some serious problems.

The dome is not impervious to electromagnetic radiation. Some interference can still get through (albeit less, compared to open space). Think about how, say, cell phones cause wireless interference: more electromagnetic radiation in the air, interfering with your network. This doesn't cause any power outages or surges, but still can take down the field network. It's similar with regard to that kind of atmospheric noise - it doesn't have to be enough for a power surge to impact the network.

There are a few problems here,wouldn't make sense that the entire stadium be electrically grounded in case of lightning striking the dome itself? Another concern would be based on the information provided why wouldn't wifi in schools or homes lose connection during bad storms? There is also the problem that the only way the noise would affect the robot connection would be if we all connected to that router since 3G and the competition signal run on different wavelengths.

[Steven Donow]

I would be curious to know where the Einstein field was during regionals. I'm treading very lightly here, because we had just come to terms with our season and accepted fate for what it is: a slippery eel. For those who just watched 118, you saw exactly what happened in the CT semis. Watching the Robonauts brick again, and again, was terribly sad. They have such a beautiful robot, and tripling with them was always picture perfect.

The field in CT was the same as the one in NYC , which also had persistent comm issues, predominantly on the red side. So, where has Einstein been?

Einstein (correct me if I'm wrong) is the emergency backup field that spends all season sitting in a warehouse in Memphis, Tennessee.

[ghostmachine360]

You are correct.

[bduddy]

I don't know if its possible or not, but after living in ST. Louis my whole life, I have seen this stuff happen during bad weather. Sometimes during bad weather I will lose wireless, without a power surge or outage. I wouldn't put it past the Midwest...The worst of the storm (hail and thunderstorms) came during the finals matches, which had the least communications problems. It's certainly possible, of course, but I don't see much of a link there.

Anyway, I think this question has to be asked: Why is the Einstein field one that has (generally) not been used at all during the season? Yes, I'm sure it's tested just as much as the other fields if not more, and I understand the appeal of playing the final matches on a pristine, unused field, but I think the benefits of using a tried and tested field outweigh those. If in a future year there are again a lot of communication problems, they could even make it part of the protocol to play the finals on the field that had the fewest such issues during its own matches.

[Doc Wu]

If only we could go back to IFI's radio system... wifi problems solved.

Yeah. Try and push a video signal over that!

[Kevin Sevcik]

Wheeee. Well I feel a need to post a small rejoinder to plnyyanks' post above. My only quibble with his post is the suggestion that this isn't anyone's fault. This is unequivocally, without a doubt, FIRST's fault.

Understand, I'm not saying that the root cause lies with the field or something. I think it's pretty clear the the root cause of these failures must be associated with particular robots in some fashion. Galileo didn't really have any problems after switching to the Red station, after all. So it's unlikely that the issue is with a particular station color or number.

Nevertheless, it is FIRST's fault even if every single comms issue can be traced back to some specific problem with a robot. This is for the simple reason that we're playing with a FIRST mandated control system on a FIRST field, following FIRST's instructions for setting up and running on the field. All the equipment and software we use is supplied through FIRST. If they don't know this stuff well enough to troubleshoot these problems and help teams fix issues, it is entirely their fault for not doing due diligence and understanding their system. Yes, yes, a team or robot might be the cause of a particular case of comms issues, but it's inexcusable if FIRST can't tell that team how to fix the problem.

(shakes cane) Back in the day of IFI controllers, we didn't have these kinds of problems. When there were comms or controls issues, the IFI people worked the problem and solved it. The back in '07, IFI moved to new radios and teams had comms issues with them. IFI identified the problems and had FIRST post a list of fixes and solutions in a Team Update. Before the Week 1 regionals. Clearly, we have been in this situation before, only back then things got fixed because there were people available that knew the entire control system from code to chips.

The situation now is obviously different. I'm certain FIRST has experts available that know the NI hardware and software, PDB, etc. just as well as IFI knew things. I'm just as certain they don't have anyone that knows anything about our communication equipment since we're using off the shelf consumer components for that vital part of the robot. Heck, it took a FIRST team member/mentor writing a program to automate WPA programming back in '10. FIRST clearly has been treating our wifi equipment as a magic black box that should just work. It's entirely their fault for doing so, and we're seeing the results of it now.

[plnyyanks]

Another concern would be based on the information provided why wouldn't wifi in schools or homes lose connection during bad storms?

This: I don't know if its possible or not, but after living in ST. Louis my whole life, I have seen this stuff happen during bad weather. Sometimes during bad weather I will lose wireless, without a power surge or outage. I wouldn't put it past the Midwest...

There are a few problems here,wouldn't make sense that the entire stadium be electrically grounded in case of lightning striking the dome itself?
I'm not saying that an electrical current (like, electrons flowing) passes from the sky, through the dome, and into the field network. Like you said, I'd assume the dome is grounded, preventing this from happening. I am saying that the culprit might be electromagnetic radiation, not current. Electromagnetic radiation exhibits exhibits wave-like behavior as it travels through space (https://en.wikipedia.org/wiki/Electromagnetic_radiation). This would allow it to travel through the (grounded) dome and interfere with the network. The wave emitted by the storm would interact with the wave emitted by the router, get into conflict, and mess things up for the teams.

[Deetman]

There are a few problems here,wouldn't make sense that the entire stadium be electrically grounded in case of lightning striking the dome itself? Another concern would be based on the information provided why wouldn't wifi in schools or homes lose connection during bad storms? There is also the problem that the only way the noise would affect the robot connection would be if we all connected to that router since 3G and the competition signal run on different wavelengths.

Agreed... If that entire stadium isn't grounded properly I wouldn't want to be in it during an electrical storm. With the proper grounding that building makes a Faraday cage for everything inside it limiting (or eliminating) any electromagnetic waves from going in or out. This is the same reason that generally your cell phone loses signal quality inside large buildings like the stadium.

Satellite TV (well any satellite communications really) have issues with electrical storms because there is no way for that signal to avoid the interference the storm provides.

Regarding cell phones and other connected devices, generally they are running on different frequencies than the 5GHz (I'm assuming they were not using 2.4GHz) WiFi we use in FRC. While harmonics can be nasty and have an effect, generally they won't be strong enough to be an issue; they come more in to play in things like Space to ground transmissions (see LightSquared interfering with GPS despite being in a different frequency band).

There is no doubt in my mind that any issues were not caused by general electromagnetic interference with the WiFi system. I'm not discounting multiple WiFi networks being present confusing the routers on board our robots, but that is not directly related.

[Anupam Goli]

Even if it wasn't interference, what happened to Code Bonde? Why couldn't they switch out to a Code Bonde setup and tried that?

[arizonafoxx]

Here are my two cents for what it is worth.

We had around an hour after every field was finished before the Einstein field started. When the finals started on Einstein we had to listen to around 50 minutes of speeches and awards before we got to the first match. Was this really the best use of time in hind sight? Why can't each robot have a practice connection during that down time. NASCAR has practice time on the track, Soccer has warm ups on the field, Football has warm ups on the field before the super bowl. With the amount of time that this field sat in a warehouse how many hours of time were robots connected to the field. Every other field at the event had a previous regional where it was used and the equipment was tested. Second why can't one match be completed before the first awards and Deans speech are presented. If there was a comms problem there would have been around 50 minutes to troubleshoot the problem while Dean and others were speaking. Don't get me wrong I like listening to the speeches to a certain extent but I also think everyone is excited and want to at least see one or two matches right away before the speeches start. Had everyone seen there was a potential problem with this field before there is an issue where they are behind schedule due to Dean's speech or something else they could have realized they needed to cut the speeches shorter to rerun a match or something.

Just my two cents.

I think there will be a valid solution to this by next year that is stated in the manual. At the very least the the teams should have switched sides so that each team had to deal with the same field issues.

[Steven Donow]

Even if it wasn't interference, what happened to Code Bonde? Why couldn't they switch out to a Code Bonde setup and tried that?

Wow. I totally forgot about Code Bonde until now. Why DIDN'T they do that? I assume there must have been a reason...has it been used at all this season?

[Anupam Goli]

I heard something about it being implementable at the regional level and not beyond that, but I'm not sure myself. Code Bonde was put in place for situations like this where wireless interference could be a problem.

[Deetman]

I heard something about it being implementable at the regional level and not beyond that, but I'm not sure myself. Code Bonde was put in place for situations like this where wireless interference could be a problem.

If wireless interference truly was the root cause, I'm sure they would have put it into effect. All of these events have a Spectrum Analyzer at them monitoring the levels at various frequencies. I'd imagine that everything on that front was well within tolerances.

Not to mention that it hasn't been used all season. Who's to say the problem wouldn't be made WORSE or you introduce other problems in the process of switching?

[plnyyanks]

If wireless interference truly was the root cause, I'm sure they would have put it into effect. All of these events have a Spectrum Analyzer at them monitoring the levels at various frequencies. I'd imagine that everything on that front was well within tolerances.

If that's the case, then my point about the weather may be moot. This is news to me.

[Kevin Sevcik]

Bonde was used once this season at FLR. Looking through the FLR thread, there wasn't much mentioned as to why. There also wasn't any comment on comms problems or anything, so go figure.

If it's implementable on a regional level, you should be able to run it on Einstein. They're clearly not running anything different on FMS at Einstein, judging by the fake Qual matches on the Match Results page. I suspect no one thought there would be a problem, and there wasn't time between matches to switch out the setup to the Bonde setup.

[Steven Donow]

Bonde was used once this season at FLR. Looking through the FLR thread, there wasn't much mentioned as to why. There also wasn't any comment on comms problems or anything, so go figure.
.

I would think it could be due to the issues at FLR last year, which I believe are why Code Bonde was created.

Here's something else to note, in regards to the field being a fresh, never used field. Prior to the matches, the red alliance stations had the numbers, "16, 2194, 330". I don't know what the blue station had because of where I was sitting, but this struck me as something very strange.

[arizonafoxx]

Bonde was used once this season at FLR. Looking through the FLR thread, there wasn't much mentioned as to why. There also wasn't any comment on comms problems or anything, so go figure.

If it's implementable on a regional level, you should be able to run it on Einstein. They're clearly not running anything different on FMS at Einstein, judging by the fake Qual matches on the Match Results page. I suspect no one thought there would be a problem, and there wasn't time between matches to switch out the setup to the Bonde setup.


I think there was enough time between matches to switch this out. I mean how many awards were given out between each match. It may have been a logistics issue of not having the equipment where it needed to be in order to implement it, but there was enough time. My guess is that since it had not been used much the whole season that they left it in the truck when they unloaded the equipment.

[JM987]

180, 25, and 16 played and won on the red alliance. The field issues seemed to be just with the red alliance, so I would not consider it a "tainted" win.

I cheered when they announced replaying both matches. That was the correct decision, and as far as I know, the only decision they could have made.

That's not true. We were on the blue alliance in the finals and were having major problems.

[Anupam Goli]

That's not true. We were on the blue alliance in the finals and were having major problems.

Ooh, we have someone from one of the Einstein teams. What were the reactions of the teams with issues, and how did the FTA look at it?

[Kevin Sevcik]

I would think it could be due to the issues at FLR last year, which I believe are why Code Bonde was created.

Here's something else to note, in regards to the field being a fresh, never used field. Prior to the matches, the red alliance stations had the numbers, "16, 2194, 330". I don't know what the blue station had because of where I was sitting, but this struck me as something very strange.Look at the "Match Results" webpage. There's a fake Quals match with those numbers on the blue alliance (behind the red goals). I guarantee you they set things up on Einstein by setting up a new regional with the division winners and finalists, scoring some matches, and running a fake alliance pairing.

[jasonbrooks]

Wow. I totally forgot about Code Bonde until now. Why DIDN'T they do that? I assume there must have been a reason...has it been used at all this season?

I'm sorry, but I am from an FTC team...what is Code Bonde?

[Anupam Goli]

I'm sorry, but I am from an FTC team...what is Code Bonde?


Code Bondé



Summary: If a competition’s wireless environment has too many Access Points (AP’s), the wireless bridge provided to teams will not be able to connect to the field’s AP. Should FIRST determine that an event meets this criteria, we will employ an emergency procedure, called Operation Bondé, to insure that the event continues with minimal impact. This determination will most likely be made Wednesday or Thursday morning, and will be communicated to teams as early as possible. In the event of an Operation Bondé, teams will use a DLink DIR-825, provided by FIRST in the queue, instead of the DAP-1522 that’s required by the rules.



Background: The DAP-1522 wireless bridge required for competition will not link to FIRST’s field access point if there are more than approximately 60 active access points in the venue. FIRST is working with all scheduled venues to reduce the number of access points active during the competition.



In the event that a venue cannot limit the number of active access points, FIRST will implement the emergency WiFi plan. FIRST has identified and tested an alternate bridge, the DIR-825, which is successful at reliably connecting with the FIRST access point in hostile WiFi environments like those described above.



Detail: FIRST will ship a small batch of these devices to each event to be used in the event of a hostile wireless environment. Teams will be asked to trade out their DAP-1522 wireless bridge for the FIRST provided DIR-825 while they’re in queuing, use it in the match, and then return it to the field crew after leaving the field.


Code Bonde is used as an emergency in case of a hostile wifi environment.

[Botwoon]

Ooh, we have someone from one of the Einstein teams. What were the reactions of the teams with issues, and how did the FTA look at it?

Teams that lost because of it were livid. Teams that benefited from it shared the opinions of everyone here. I thought that the Curie eliminations were far more exciting than any single match played on Einstein. The finals on Curie were intense.

[ErikEdhlund]

This:


I'm not saying that an electrical current (like, electrons flowing) passes from the sky, through the dome, and into the field network. Like you said, I'd assume the dome is grounded, preventing this from happening. I am saying that the culprit might be electromagnetic radiation, not current. Electromagnetic radiation exhibits exhibits wave-like behavior as it travels through space (https://en.wikipedia.org/wiki/Electromagnetic_radiation). This would allow it to travel through the (grounded) dome and interfere with the network. The wave emitted by the storm would interact with the wave emitted by the router, get into conflict, and mess things up for the teams.

But the problem is that based on the theory behind electromagnetic radiation we would have interference during ever major thunderstorm in lesser insulated buildings. Also if there was any major waves that would affect the stadium, other electronic devices such as phones and laptops would be affected as well.

[JM987]

Ooh, we have someone from one of the Einstein teams. What were the reactions of the teams with issues, and how did the FTA look at it?

I didn't come on to rant just recognize something went wrong. If there were Com issues involved they should've took a better approach instead of in a sense saying oh, well better luck next year. I mean, what was the point replaying SF1-1 and SF2-1 then?

As far as the FTA goes, it seems like they didn't do anything to change the outcome or else it would've never happened. Or at least waited until the issue was resolved.

[Steven Donow]

But the problem is that based on the theory behind electromagnetic radiation we would have interference during ever major thunderstorm in lesser insulated buildings. Also if there was any major waves that would affect the stadium, other electronic devices such as phones and laptops would be affected as well.

Well, for what it's worth, as I was leaving the dome after Einstein, I was unable to send texts on my phone and got the error, "Network not responding". I have Verizon and not a smartphone, if this post even matters at all haha.

[Deetman]

Well, for what it's worth, as I was leaving the dome after Einstein, I was unable to send texts on my phone and got the error, "Network not responding". I have Verizon and not a smartphone, if this post even matters at all haha.

You and everyone else trying to leave were using their phones, inside a pseudo Faraday cage, all trying to communicate with the same few cell towers. No surprise at all that the cell network could have been overloaded.

[Steven Donow]

You and everyone else trying to leave were using their phones, inside a pseudo Faraday cage, all trying to communicate with the same few cell towers. No surprise at all that the cell network could have been overloaded.

I assumed it was something like that, as I said, that was a pointless post I made haha.

[splatter]

in my opinion no one really won nationals. those disconnects cost the alliance the final matches. this could have changed the whole out come of the entire event. you should not pay upwards of 10,000 plus dollars and be left in the dust. they should have taken a 1 hour time out and finished the awards. and then finished the matches. leaving those teams behind is not the spirit of gracious professionalism at all.
that's just my too cents on the issue

[DominickC]

I completely agree, splatter. The amount of issues with the FMS was unprecedented, especially given that many of the robots have had nearly no FMS issues until Einstein.

With the problems going on, they might not have been able to do much of anything in an hour. I don't know how long it would've taken to enact Code Bonde, but it should've been tried.

[splatter]

our team had connections problems all day luckily we were able to work them out but the FTA did every things in his power to help us he did not leave us flat once. and that was in practice matches and early qualifiers. so it was very disappointing i really felt like no one truly won at all in a general sense. but nice job to all of the teams in the finals nice job all ! to find a true winner it should all be re done but it wont happen. good job winners!

this is a reply to

I didn't come on to rant just recognize something went wrong. If there were Com issues involved they should've took a better approach instead of in a sense saying oh, well better luck next year. I mean, what was the point replaying SF1-1 and SF2-1 then?

As far as the FTA goes, it seems like they didn't do anything to change the outcome or else it would've never happened. Or at least waited until the issue was resolved.

from jm987

[efoote868]

That's not true. We were on the blue alliance in the finals and were having major problems.

Like I said, "seemed to be". The only bots that were dead on the field were on red.
From the stands, it's difficult to tell when a team is having comm issues, unless they stay put through the entire match. :(

I think FIRST needs to take a good hard look at their system, and find a way to implement something more robust. Perhaps the field should remain pristine, but use a different (used) FMS.

[splatter]

from what i remember the robot status light (the orange blinking light)
was in the non connection state of the non moving robots.

[Racer26]

Like I said, "seemed to be". The only bots that were dead on the field were on red.
From the stands, it's difficult to tell when a team is having comm issues, unless they stay put through the entire match. :(

I think FIRST needs to take a good hard look at their system, and find a way to implement something more robust. Perhaps the field should remain pristine, but use a different (used) FMS.

233 (Curie, blue) was dead in F1.

[efoote868]

233 (Curie, blue) was dead in F1.

Missed that. Thanks for the correction.

[slijin]

Like I said, "seemed to be". The only bots that were dead on the field were on red.
From the stands, it's difficult to tell when a team is having comm issues, unless they stay put through the entire match. :(

I think FIRST needs to take a good hard look at their system, and find a way to implement something more robust. Perhaps the field should remain pristine, but use a different (used) FMS.

The first part is incorrect. A flashing light at the corresponding alliance station indicates loss of communications. It was in this way that I and other mentors on my team observed comms issues. For me, at least, it was extraordinarily clear that the entire Curie alliance suffered from comms issues at some point during finals. The only two robots to maintain comms in finals, I believe, were Bomb Squad and Raider.

A different FMS is an option, albeit a logistically impractical one. The better solution, imo, to extend on Kevin's post, would be for FIRST to at least have demonstrated throughout the season (via team updates or some manner) that they were actively working to isolate and resolve the issue. It also should have been clear that pre-match testing was absolutely imperative given the prevalence of these issues throughout the season.

FIRST's unwillingness to explain these issues was also unforgivable.

[DominickC]

FIRST's unwillingness to explain these issues was also unforgivable.

This, I agree with.


A different FMS is an option, albeit a logistically impractical one. The better solution, imo, to extend on Kevin's post, would be for FIRST to at least have demonstrated throughout the season (via team updates or some manner) that they were actively working to isolate and resolve the issue. It also should have been clear that pre-match testing was absolutely imperative given the prevalence of these issues throughout the season.

This, I do not. If the FMS is at the root of the issue, why is it remaining in place? Would you solve the issue of termites creating lots of sawdust in the kitchen by buying a better vacuum?

If the TCP/IP protocol is to blame, it's time to move on to a more reliable technology.

[MrTechCenter]

I think most if not all problems that occurred could have been prevented if the connections on Einstein were tested earlier in the day. If the problem had indeed been the field, they could've spent the rest of the time up until the finals troubleshooting. If there were no problems before but problems later, the bots should've been check then to see if it was the bot's fault. If not, they definitely should have looked into radio interference and storm interference which they had the equipment to do so (at least for the radio part anyway). I don't think it was so much as an inability to find the issue as the lack of will and patience to do so.

[Deetman]

If the TCP/IP protocol is to blame, it's time to move on to a more reliable technology.

TCP/IP as designed and properly implemented is a reliable protocol. If a packet is not received properly it will be resent. Is it right for FIRST? Is the implementation throughout the system correct? I don't know.

[Yankeefan181]

Something to keep in mind:

118 has had communication errors a couple times this year. Yes, there were clearly problems on Einstein that effected a lot of teams. However, once they worked on the problems and replayed the matches, it was only 118 that never moved. I don't think we can blame the entire thing on FIRST, especially since we don't have all the details.

[goldenglove002]

In regards to the weather possibly playing a factor into interference with the WiFi:

I'm by no means a radio-frequency or weather expert so I just had a conversation with some guys in the meteorology department at my school and they were skeptical about the storm being a factor. Lightning strikes do produce radio waves called sferics which would make their way into the building, however they are usually in AM frequency range, and shouldn't cause interference with the devices that FIRST uses on the field.

They participate in storm chasing during the spring and use plenty of wireless networks during the process, so they have plenty of experience with wireless devices in the middle of large thunderstorms. While I'm sure there are some equipment differences, they haven't ever had interference problems in the middle of a storm, nor heard of anyone else having those issues. Of course this isn't a 100% comparable situation, but it's relatively close IMO.

[Lewis Ruskin]

It might help to switch to the 5 gz bandwidth for wifi. Some laptops would need to have an adapter for the network. Very few devices use the bandwidth and it would cut down on interference.

[arizonafoxx]

For what is worth. One thing I was told by the FTA at an off season event. The signal to start Autonomous or Teleop is only sent once at the beginning of that state. I personally have no idea how the field communicates with the robot but this statement makes sense for why some robots don't move. If you only send a signal once and in that split second while that signal is active there is a glitch in the WiFi (which happens) then the robot misses the signal and won't ever activate that particular state. However this is not how TCP/IP is meant to work. If a signal is missed there will be a message to send the signal again for a few more attempts to help create the full packet. It seems to me that this is not happening. When a robot is not communicating with the field there are separate signals being sent to flash lights above the driver station. Why can't a simple protocol be created that while this light is flashing there are multiple attempts to tell the robot what state to be whether it be Auto or Teleop. Having a one shot attempt to tell the robot to go seems like it is leading to these problems.

How much do we know about how the field works? Can this hypothesis be proved invalid?

It also seems that when the FTA is looking at packet transfer there are packets that tell the robot to be disabled till the match starts and then there are packets after the match starts that have the data from the DS but how many packets tell the robot to switch from disable to Auto and then to Teleop. When there are only a few packets missed this does not seem like a lot to the person watching the graphs and what not on the FTA table but its not about how many packets are lost its about which packets are lost. Does the FTA have a way to tell that the packet to switch from disable to Auto and Teleop has been received properly? Can the robot tell the field that it has received the packet properly so a status flag can be used to light and led on there comms station? I agree that telling a team they have no idea what is happening is not acceptable. Anytime you say you have no idea what is happening it just means you are either A) not taking enough data points or B) you are not taking the data points on the right data.

[Racer26]

@Ruskin: we DO use 5GHz for the robots.

I propose a move to the licensed 3GHz 802.11 band.

[efoote868]

Does anyone know which frequency bands the robots use? Could weather radar be messing them up?

[akoscielski3]

Is it possible that since Einstein was the back-up field that if there was an Update on the FMS for every-field, Einstein Never got that update, thus causing all these Robots to have a problem?

Proposal for IRI

All alliances come to IRI and we play the Real Einstein matches to try and find who the winner should be?

[Racer26]

Is it possible that since Einstein was the back-up field that if there was an Update on the FMS for every-field, Einstein Never got that update, thus causing all these Robots to have a problem?


well there's an idea nobody I saw thought of before.

[Racer26]

Does anyone know which frequency bands the robots use? Could weather radar be messing them up?

5GHz 802.11a WiFi

[BJC]

While this problem has been brought to light by the Einstein matches I do not think it is new. At MSC 33 died in the semis about 3 seconds into teleop. In the finals 67 had similar looking drop outs although I cannot confirm that it was in fact an identical problem. 1114 also seemed to have issues although those could be on the robot side (Both 67 and 1114 were connection issues.) Perhaps we, as a community, should begin documenting all of these failures throughout the season in order to identify patterns and the actual quantity of failures. It is easier to accept there is a problem when you have data to back it up and with many examples of failure to look through it would hopefully make identifing the problem that much easier.

Regards, Bryan

[goldenglove002]

Does anyone know which frequency bands the robots use? Could weather radar be messing them up?

Weather radar uses microwave radio pulses that would not have any effect on other wireless signals. Otherwise your internet would drop every time the radar makes a rotation (radars run at all times, not just when there is a storm)

[efoote868]

5GHz 802.11a WiFi

There are several channels / frequency bands.
E.g. 36, 40, 44 (5180, 5200, 5220 mhz). If they use the middle channels (unlikely, but you never know), the radios are required to listen for radar and switch channels (called Dynamic Frequency Selection).

[BHS_STopping]

Does the cell phone noise hypothesis hold any water? I feel like these things happen when there are a lot of people near a field, such as during elimination matches (see 1717 on Newton for similar problems). A lot of phones including mine have WiFi capabilities and may have interfered with the routers on the field.

[BrendanB]

Something to keep in mind:

118 has had communication errors a couple times this year. Yes, there were clearly problems on Einstein that effected a lot of teams. However, once they worked on the problems and replayed the matches, it was only 118 that never moved. I don't think we can blame the entire thing on FIRST, especially since we don't have all the details.

I was at the event where 118 was having issues (CT) and it was traced back to their robot.

Whatever this was really cost the championship for every alliance. They all were strong and each could have taken it if they were running on full power. 987's alliance was extremely close in the last match the High Rollers only needed to score a few more but they were motionless.

[Kevin Sevcik]

Does anyone know which frequency bands the robots use? Could weather radar be messing them up?For complete thoroughness, specs on NOAA's NEXRAD radars say they use 2.7Ghz to 3.0Ghz. NOAA also says they're constantly emitting high energy pulses. Short high energy pulses. Something like 1000 pulses per second, but the transmitter is still only active about 7 seconds out of every hour. So it seems unlikely that that's the culprit.

[philso]

Am I correct in thinking that the radios use spread-spectrum techniques? It has been quite a few years since I have worked with such radios.

[avanboekel]

Okay, somethings wrong here. In the last 4+ hours since championships have been over, there have been 14 posts in the 'congrats 16, 26 and 180' thread. There have been hundreds complaining about the field issues. Whether you like it or not, they are the FIRST World Champions. It isnt their faults that there was a problem with the field. Their achievements are just as important as field issues. I suggest that everyone heads over to the thread to congratulate them.

http://www.chiefdelphi.com/forums/showthread.php?threadid=106040

[Racer26]

There are several channels / frequency bands.
E.g. 36, 40, 44 (5180, 5200, 5220 mhz). If they use the middle channels (unlikely, but you never know), the radios are required to listen for radar and switch channels (called Dynamic Frequency Selection).

Holy crap. I didn't know about that.... I think that's part of the source of the problems.

I'm pretty sure FMS sticks each robot on a different 5GHz channel, and actively manages them, in a way that *may* not be compatible with DFS.

If the consumer grade d-link detects 5GHz radar stuff in its channel, and decides to try to jump onto another channel and FMS doesn't play nice on that decision guess what? boom, robot dead.

Would also explain why Red 2 in particular was having issues. Guessing the closest radar antenna was on the frequency that Red 2 was arbitrarily assigned by FMS.

[superbotman]

I have witnessed competitions as a FRC competitor and as a scorekeeper at a regional.
I have spent some time thinking about what they could do to fix/prevent the problems occurring. First, they should have used an electronics system from a field that has been through regionals and has proven to work in a competition.
Then, they should have tested robot connections before matches, possibly going as far as to run a match with the robots sitting on their carts, not on the field, and the sounds turned off so the audience doesn't hear.

If they didn't get solved with the earlier ideas, they should have started changing the field electronics out incrementally starting with the portions that interface with the robot, and prepping a second server to try as a complete overhaul and start the electronics from scratch. Then they should have tested connections during awards and speeches.

Replaying the first two matches was the right thing to do, but when problems persisted, they should have tried to do more, if for no other reason - because this is EINSTEIN.

[efoote868]

For complete thoroughness, specs on NOAA's NEXRAD radars say they use 2.7Ghz to 3.0Ghz. NOAA also says they're constantly emitting high energy pulses. Short high energy pulses. Something like 1000 pulses per second, but the transmitter is still only active about 7 seconds out of every hour. So it seems unlikely that that's the culprit.

This was what I was thinking of:
http://en.wikipedia.org/wiki/U-NII
Where radar avoidance is mentioned several times.

[BrendanB]

I have witnessed competitions as a FRC competitor and as a scorekeeper at a regional.
I have spent some time thinking about what they could do to fix/prevent the problems occurring. First, they should have used an electronics system from a field that has been through regionals and has proven to work in a competition.
Then, they should have tested robot connections before matches, possibly going as far as to run a match with the robots sitting on their carts, not on the field, and the sounds turned off so the audience doesn't hear.

If they didn't get solved with the earlier ideas, they should have started changing the field electronics out incrementally starting with the portions that interface with the robot, and prepping a second server to try as a complete overhaul and start the electronics from scratch. Then they should have tested connections during awards and speeches.

Replaying the first two matches was the right thing to do, but when problems persisted, they should have tried to do more, if for no other reason - because this is EINSTEIN.

This is what sums up my feelings. You could tell that what happened on the field set the tone for the arena, the announcers weren't the same, Dean, Woodie, etc weren't the same and we were just seeing them via a webcast. The fact that nothing was done to remedy the problem is flabbergasting when you are in the most important matches of the season.

118 might be a seperate issue, but you don't have robots just start to lose comms in matches all at once. I could see 1114's issues come back but have robots that performed 100% in their division elims hit Einstein and all but three experience issues you have a MAJOR problem on your hands.

[Deetman]

Pretty sure our radios operate in the 5.15-5.25GHz range. From D-Link's website:
Due to government regulations the 5.25~5.35GHz and 5.47~5.725GHz wireless bands are removed.

As a result, per U-NII our radios are not subject to DFS.

[efoote868]

Pretty sure our radios operate in the 5.15-5.25GHz range. From D-Link's website:


As a result, per U-NII our radios are not subject to DFS.

In the future, it may be advantageous to operate in that band, specifically because most consumer grade wifi does not.

[Bryan Herbst]

TCP/IP as designed and properly implemented is a reliable protocol. If a packet is not received properly it will be resent. Is it right for FIRST? Is the implementation throughout the system correct? I don't know. Blaming TCP or IP should be completely out of the question. First of all, these are two of the most (really, THE most) used networking protocols in existence today. They are used by every single device (that's billions of devices) that connects to the internet.

Furthermore, one of TCP's advantages over other transport-layer protocols (such as UDP, commonly used for tasks such as video streaming) is it's reliability.

Which brings me to...
For what is worth. One thing I was told by the FTA at an off season event. The signal to start Autonomous or Teleop is only sent once at the beginning of that state. Though I don't know for sure, I'm assuming that the FTA was telling you that the application only sends the signal once for auto/teleop, constantly for disabled. Even though the application layer only sends one message down to the lower layers (TCP, IP, etc), TCP will keep sending the packet over and over again until it receives an acknowledgement. If that packet is dropped, it will be resent.

Is it possible that since Einstein was the back-up field that if there was an Update on the FMS for every-field, Einstein Never got that update, thus causing all these Robots to have a problem?
I very highly doubt that. I also doubt that never having used the Einstein field before would be to blame for any issues. The hardware and software are identical to that used on every other field.


Are there kinks that need to be worked out? Yes.
Do I think it is the FMS or field hardware's fault. Not really.
Do I think it is the robots' fault? Not really, though I have seen enough robots to believe that robots with various electrical or programming issues are just confuddling this problem more.

[Walter Deitzler]

Okay, somethings wrong here. In the last 4+ hours since championships have been over, there have been 14 posts in the 'congrats 16, 26 and 180' thread. There have been hundreds complaining about the field issues. Whether you like it or not, they are the FIRST World Champions. It isnt their faults that there was a problem with the field. Their achievements are just as important as field issues. I suggest that everyone heads over to the thread to congratulate them.

http://www.chiefdelphi.com/forums/showthread.php?threadid=106040

This.

[Kevin Sevcik]

This was what I was thinking of:
http://en.wikipedia.org/wiki/U-NII
Where radar avoidance is mentioned several times.Well the only problem with this theory is that DFS is an Access Point feature. The Access Point picks a channel to transmit on based off its detection of possible interference. I don't know exactly how the FMS manages the wireless access points, but I think the wireless channels are manually selected. Even if they're not, the DFS wouldn't kick in that often, and the WiFi clients shouldn't have trouble following the AP from one channel to the next. But mostly, I think FIRST is setting channels manually so this wouldn't be a problem.

[Chris Fultz]

I think FIRST needs to take a good hard look at their system, and find a way to implement something more robust. Perhaps the field should remain pristine, but use a different (used) FMS.

I think only the field and game elements are pristine, and the electronics have been used at other events.

[superbotman]

I think only the field and game elements are pristine, and the electronics have been used at other events.

I don't know for certain, but Einstein is a complete field and as far as I know they use that field in its entirety for those matches.

[Kris Verdeyen]

I was at the event where 118 was having issues (CT) and it was traced back to their robot.

Don't have anything to add except to say that this is not true. The problems in CT were never diagnosed. We swapped out a cRio in Houston because we saw the same thing in a practice match, and couldn't bear the thought of doing nothing, even though the cRio swap didn't make much sense. We had no problems after that swap until what you saw on Einstein.

[PayneTrain]

I think only the field and game elements are pristine, and the electronics have been used at other events.

Bill has always implied that the emergency field in Kentucky (?) or Tennessee (?) was a total emergency field in case something ever happened to another one; otherwise it was just for Einstein. However, I would consider an electronics system that was in use for a few events to be more solid than an untested one... at least we know where and when it has and hasn't worked before.

I imagine/blindly put faith into the idea of it will be an awkward topic of discussion in a blog post from the director.

Also, while I will congratulate the teams that have one (and they do deserve it) I believe it's just as important, if not more, we solve the issues relative to the complex and mentally/emotionally taxing 4-year-old communication system.

[techhelpbb]

Okay, somethings wrong here. In the last 4+ hours since championships have been over, there have been 14 posts in the 'congrats 16, 26 and 180' thread. There have been hundreds complaining about the field issues. Whether you like it or not, they are the FIRST World Champions. It isnt their faults that there was a problem with the field. Their achievements are just as important as field issues. I suggest that everyone heads over to the thread to congratulate them.

http://www.chiefdelphi.com/forums/showthread.php?threadid=106040

I agree. These communications problems are not new. They've persisted through out all levels of competition to this point.

As I've stated elsewhere when Team 11 had issues 3 times in Philly. I didn't feel that it dramatically effected our placement based on what we saw when we competed entirely free of issues at other events. By a turn of events we went to Championships anyway.

Sure there's room to argue the point. However, as long as the people are effected fairly randomly let's not forget that these folks built some really phenominal robots and we all played the same game.

Chance is always an element. Be it a bizarre unexpected balance, a sudden broken part or even a software failure. Sometimes the chance is a spectacular show and sometimes not.

As far as how FIRST handled this? Troubleshooting of this type has been limited to the events. Everything else, like this survey, is generally a collection of anecdotal evidence. I expressed my concerns about field communications before this year's first event (based on our off-season experience at Monty Madness last year...see the topic about alternate control systems) and I expressed my concerns about how the troubleshooting was pushed back generally at all events in the 2 recent years with this radio system in the other topic. I myself was spare parts at the MAR Mount Olive events and we had a few issues with this problem and despite much effort to track the only thing that helped them was driving with the camera off and hence mostly at a disadvantage. The situation was put on this path by FIRST and people close to FIRST. This was the risk at every event and every event had the same problem not just the Championship. I presume that FIRST had figured they had this problem well under control when they started Championship given the pattern of the behavior. Perhaps FIRST just felt it fair not to eliminate the problem now that it had already messed with all the events up to that point...but they did allow some replays which is inconsistent (perhaps the story changed under the hot lights). Things apparently did not work out. I saw the look on everyone's face. They don't need this topic to tell them this can not be allowed to happen again....we need quantifiable evidence not anecotes and we need it when the problems happen not after the fact when things are all apart or it's just speculative no matter who's speculating really. We as teams can't test on their competition fields; both because we don't have access to them all the time and because often the rules prohibit the effort during competition matches.

I'm sure it'll get fixed it's hard to ignore it's a priority under the circumstances.

[EricH]

What I would do with the Einstein field is to run it in a Week 0 event before putting it into storage. It won't be as pristine, but you get a chance to clean it up before the big show, and you know that it worked (or didn't) when you put it in the trailer. I'd rather have it working and slightly ugly than pretty and not working.

[arizonafoxx]

What I would do with the Einstein field is to run it in a Week 0 event before putting it into storage. It won't be as pristine, but you get a chance to clean it up before the big show, and you know that it worked (or didn't) when you put it in the trailer. I'd rather have it working and slightly ugly than pretty and not working.

Another idea relating to this. Take the electronics from the most reliable field. AKA has the least comm failures throughout all the regionals. But take the carpet game pieces and field elements from the backup field. Then you have the best comms with the best looking field. The backup electronics could be used at MAR or another state district to get them ready for CMPs. Maybe not the best solution for the backup electronics but it better than them being tested at CMPs on Einstein.

[PayneTrain]

What I would do with the Einstein field is to run it in a Week 0 event before putting it into storage. It won't be as pristine, but you get a chance to clean it up before the big show, and you know that it worked (or didn't) when you put it in the trailer. I'd rather have it working and slightly ugly than pretty and not working.

We would all also love Einstein with maybe a few short speeches.

Also, I will state one more thing before I turn off for tonight.
I came into FRC as a freshman. You know what my first memory of a regional was? Not starting play until the afternoon in 09 because the field was bugging out. You know the thing a remember most vividly in 2010 outside of our ludicrous finalist bid and the case of the missing in-game piece? Standing in queue for 45 minutes in VA while we dealt with field issues. 2011? Watching a robot get pulled off because of communication issues the field was experiencing (the station where the team was located had problems all afternoon with a variety of robots). 2012? Almost losing in the semifinals because the robots would lose communication midmatch and we would reboot midgame to stand a chance.

I know chance is a factor in everything, but when I pick up a remote, I don't plan on blowing up my TV. The chance of that happening is severely mitigated by evolved technology.

[Coach Tom]

I don't know for certain, but Einstein is a complete field and as far as I know they use that field in its entirety for those matches.

Was the same system used on Einstein last year (Logomotion)? If I remember correctly, during the last two matches of Logomotion a RED Alliance robot worked in autonomous and then did not work at all during driver control. Was this a robot problem?

[philso]

The radios use a spread spectrum technique for transmitting the data at the most fundamental level. Various techniques are used to enhance the data throughput from what is achieved in the 802.11a,b,g to get the 802.11n data rates.

One of the benefits of using spread-spectrum techniques is the very high noise immunity giving a highly reliable connection. More reliable than narrow band transmission techniques. Instead of transmitting a signal over a narrow bandwidth, the information is spread over a very wide bandwidth, at a relatively low amplitude. The receiver de-spreads the received signal, restoring the original narrow bandwidth and high amplitude. Narrow-band interference sources (radar, noisy electrical equipment) would be squashed in amplitude and spread over a wide bandwidth by the de-spreading process in the receiver and the majority of the noise energy can be filtered out very simply. That is one of the reasons why this technique was used by military radios for many years before the ISM band was made available for civilian use.

Due to how the spread-spectrum transmission technique works, it is difficult to believe that only one or two channels can be corrupted for a whole match by a narrow bandwith, short duration transmission like the weather radars or wideband weather phenomena such as lightning. They ought to affect all channels in a similar way, simultaneously or not at all. The only way I can think of to adversely affect a single channel would be to jam it using a spread-spectrum transmitter set to the appropriate channel. Either FIRST missed some signal source when setting up the fields or someone was intentionally jamming. If find the latter hard to believe.

The radios are a consumer grade item but their design would have been tested very thoroughly by the manufacturer and their other customers. I would imagine that the teams experiencing trouble would have been able to check for connection before each match or they would have raised a concern and not allowed the match to start.

This leads me to ask if the FMS software that manages the robots through the 802.11n link may have some bugs in it. I am sure that the developers of this software would have tested it to the best of their abilities. However, there can be issues and conditions that only show up with certain combinations of hardware and other software. The FMS software would have far fewer users and instances of use, overall, than even low-volume commercial software so obscure bugs can be very difficult to trouble shoot and would probably require tools not available to the FTA at a competition. Perhaps the developers should visit one of the upcoming off-season events and bring all their tools and toys. That is what our engineers at work do when a customer experiences unusual behaviour that the Field Service people cannot fix.

At Alamo, one of the robots in the alliance opposing ours was immobile for the whole match leading to the match being replayed. If my memory serves me correctly, it was announced that 148 had become indicated as a no-show so their channel was shut down, rendering them immobile for the duration of the match. I have no way of knowing if our (148's) experience is at all related to what was seen at the CMP but the symptoms look similar. By the way, one of the other teams did all the scoring and handily beat our alliance single-handed :(

[Chickenonastick]

Did the thunderstorms supposedly start around the same time as the beginning of Einstein?

[EricH]

One other thing that I'd like to say.

Einstein this year was a minor speedbump compared to another (http://www.chiefdelphi.com/forums/showthread.php?t=84353) event (http://www.chiefdelphi.com/forums/showthread.php?t=84605) a couple years back. Admittedly, said event had a number of factors besides the field issues complicating things.

I would not say that we've had nothing but problems since switching the control systems/wireless. However, I will say that field connection problems seem to have become more prevalent, and if this continues to be an issue, I expect that FRC HQ will be under significant participant pressure to change something with the system. Whether it's the FMS (which was mostly reliable the last couple of years), or a "Not my system" attitude held by both teams and field crew on occasion, or the routers on the robots, something needs to be made more reliable.

With the IFI system, they could tell right away whether it was the field or your robot, and if it was your robot they'd help you find and fix the problem. I haven't seen that type of full-system support the last few years--maybe because it's a multiple-manufacturer system. The system as it is is pretty good... but with this public of an issue, it could use some improvement.

[steverk]

I personally have no idea how the field communicates with the robot but this statement makes sense for why some robots don't move. If you only send a signal once and in that split second while that signal is active there is a glitch in the WiFi (which happens) then the robot misses the signal and won't ever activate that particular state. However this is not how TCP/IP is meant to work.

I think this is a very plausible theory. TCP is a "reliable" protocol, which means it retransmits afet some time period if the packet is not acknowledged. However, it only does this for unicast (point A to point B comms) packets. But multicast (point A to many points) or broadcast (point A to everyone) do not have acknowledgements and don't retransmit.

Does FIRST use multicast or broadcast in this application? I don't know, but it makes some sense that they would. By using one of these, you can be sure that nobody gets started before the others, assuming everyone gets the packet in the first place.

[Chi Meson]

Don't have anything to add except to say that this is not true. The problems in CT were never diagnosed. We swapped out a cRio in Houston because we saw the same thing in a practice match, and couldn't bear the thought of doing nothing, even though the cRio swap didn't make much sense. We had no problems after that swap until what you saw on Einstein.

I'm glad to hear you state this. This is what I had heard through the channels, but I had yet to hear it firsthand. I recall the announcement at CT, that there was "no way that the field could have caused a robot's cRIO to reboot." I subsequently talked to about a dozen engineers who said (while never saying "that's not true") words to the effect "there's no way you can make such a blanket statement as that."

I'm split when it comes to answering the OP question. There is not much else the field administrators could have done. To put it as simply as possible, there IS an issue. The issue has affected teams that have the BEST software and engineering minds. The problem/s remain elusive and diagnoses have been numerous and varied.

Administration needed to make a decision, and they did not have all day to do so. It looked like a no-win decision.

Robonauts, we all ached for you. We hope we get to see you triple-balance at least one more time!

Let's not forget to congratulate the champions.

[sgreco]

I would blame the preparation more than anything. There's no excuse not to have the field ready for FRC use. I'm more curious than anything to see what FIRST has to say: what the problem was; if they diagnosed it; why they didn't move to another field; why they replayed the first few matches, but then one after that; if the weather impacted their decisions; etc.

I'll give them the benefit of the doubt until I hear an explanation, but I really hope it's a good one. I think FIRST owes a good explanation to all the teams, volunteers and spectators.

[JJackson]

Was the same system used on Einstein last year (Logomotion)? If I remember correctly, during the last two matches of Logomotion a RED Alliance robot worked in autonomous and then did not work at all during driver control. Was this a robot problem?

We linked this problem back to the case shorting BB-775 that we were using for our arm. However last year during qualifications on Archimedes there was a blue driver station that was giving teams communications problems throughout the day ... us included.

[MarkoRamius1086]

I believe we need to go back to the videos, the full and unclipped videos, if such things exist, and look and see exactly who had issues and when they had issues. I was not there and the webcasts were focusing only on the robots that were functioning, so attempting to see who was not functioning was... near impossible.

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division? I have been hearing of definite issues with something on the 118 scene, and slight hints at something with 1114.

Bryan Culver:
While this problem has been brought to light by the Einstein matches I do not think it is new. At MSC 33 died in the semis about 3 seconds into teleop. In the finals 67 had similar looking drop outs although I cannot confirm that it was in fact an identical problem. 1114 also seemed to have issues although those could be on the robot side (Both 67 and 1114 were connection issues.) Perhaps we, as a community, should begin documenting all of these failures throughout the season in order to identify patterns and the actual quantity of failures. It is easier to accept there is a problem when you have data to back it up and with many examples of failure to look through it would hopefully make identifing the problem that much easier.
Regards, Bryan


Absolutely true and beautifully said. The first step is who, what, and where. The details, not the speculation.

[rainbowdash]

This.

Yes, the issues on Einstein were horrible, but after the first replay all you could really do is watch the train crash happen. The single most important thing is how FIRST addresses the problem in the coming weeks and months. If there isn't any comment or statement out of FIRST within a week or two, then I think we should all be highly concerned.

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.



Honestly, I would rather compromise time for a better, more fair matches.

[JesseK]

Here's a minor suggestion that may help test Einstein in the future:

Teams that are eliminated early in QF's from their respective divisions could go do a 'test run' match with some stipulations (stay off the bridges so they stay shiny, the balls may be horribly scuffed, etc). That way Einstein field can be tested with 32 different setups, including those with camera traffic, custom UDP sockets, and "nothing but dashboard" screens.

[Gregor]

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division? I have been hearing of definite issues with something on the 118 scene, and slight hints at something with 1114.





It appeared that during SF 2-1 or 2-2, each robot on the 2056 lead alliance had some time when they were unable to operate, whilst each robot was able to move at other points in the same match.

[Dad1279]

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

They could have run a match swapping red & blue. That may have helped debug, and identify if it was a field or robot issue. They could have also asked that the teams involved ship their robots back to HQ instead of home, so the engineers could try to recreate the failure and set up the Einstein hardware with these robots again.

Now that the field is broken down, and the robots that were involved are gone, all that they can do is look at logs and wonder if they have the pertinent data.

[Duke461]

Not trying to be condescending here or anything, but I didn't see any representative of any team so far post what they personally tried to do to remedy it.

During one of my team's matches, we were motionless, but after we restarted the cRIO during the match, we got comm's and code and were perfectly fine.

Did any of these Einstein teams (esp. 118) restart/reboot their cRIO during the match?

-Duke

[Culvan Van Li]

I think they did the best they could in the time they had.

I would've liked to have seen them move to matches on Einstein sooner. That could've given them more time to troubleshoot the problem by putting more of the speeches and awards in between matches.

I'm amazed that this seemed to be a mystery. I work on telecommunications equipment. Part of my job is analyzing logs of communications failures. Those logs will usually pinpoint problems pretty well. Perhaps that's just because the FCC has set the bar so high. Compared to a worldwide telecommunications network the setup for the field shouldn't be that complicated. It seems to me that FIRST probably needs to add more troubleshooting and monitoring capabilities. The good news is that they control the software in the field, they control enough of the software in the robot, and they control enough of the software in the driver's station to do quite a bit. I think they probably need to add some more capabilities inside the robot, like have it log the battery voltage for the entire match inside the cRio. Ideally the system would be able to detect common robot failures that are completely unrelated to the field. That could really help improve the quality of the "it's not the field" message given to teams. I think we all want to see the best quality matches. I wish I could help more.

Andy

[steverk]

This was what I was thinking of:
http://en.wikipedia.org/wiki/U-NII
Where radar avoidance is mentioned several times.

Good thought, but the Robonauts practice shop is adjacent to a major airport.

Further, they are also within a few miles of the NWS office in League City Texas, which has a doppler radar. That's got to be closer than the radar that covers the Edward Jones Dome.

Therefore, I think it is unlikely that they would not experience problems in their practice building, but would at the dome.

Does anybody really think the Robonauts would allow such problems to go unresolved if they could duplicate them in their shop?

[Holtzman]

Not trying to be condescending here or anything, but I didn't see any representative of any team so far post what they personally tried to do to remedy it.

During one of my team's matches, we were motionless, but after we restarted the cRIO during the match, we got comm's and code and were perfectly fine.

Did any of these Einstein teams (esp. 118) restart/reboot their cRIO during the match?

-Duke

How exactly would we go about rebooting our crio mid match when we have no communication with our robots? Walk out on the field and cycle the main breaker?

We will share our full thoughts and experiences in the coming days.

[steverk]

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.

[Brandon_L]

were not caused by general electromagnetic interference with the WiFi system. I'm not discounting multiple WiFi networks being present confusing the routers on board our robots, but that is not directly related.

I may be a bit late on this (I only read up to like page 4 so far, I've been on a 15 hour bus ride back.) but I did check the wifi signals in the dome while Einstein was going on. I only picked up about 4 signals, only like one or two were somewhat strong.

Also one question that arose on the ride back from a few people: Why would the teams agree to play on a broken field?

[RyanN]

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.

I was sort of thinking about that earlier...

Back when I was in high school, I was giving a demo at a basketball game. The score board was programmed wirelessly from a table on the side of the court.

When I drove the robot in front of the table, the robot died... I had to push it out of the area of that table, and it regained control.

This was using the IFI system, on the 900MHz radio, but it seems like everyone is moving their stuff to 2.4GHz nowadays.

Good thought!

[jblay]

FIRST did what they have done for four years when these kind of issues have occured. Give maximum one round of replay and then proced with the matches regardless of the result.
These actions are not fair to the teams on the field, winners and losers, it is unfair to the teams that competed and have already been eliminated, but most importantly it is not fair to the kids who worked so hard all season just to have it come to an end like that.
As great it is to say that we use this cool crio from NI, it is not worth this. I fear that spreading the word of FIRST has become the word of FIRST instead of teaching kids life lessons. What happened on Saturday was a result of a decision made by FIRST 4 years ago that was not in the spirit of FIRST.
It is my opinion, and I know I am not alone, that it is time for a serious change to the control system because enough is enough. It is also my opinion that FIRST needs to remedy what happened on einstein by awarding the championship to all 12 teams on einstein, but this is something that I don't think there is a chance they will do.
A big move needs to be made by FIRST here and if it isn't done I suspect much fallout in the frc community with more teams just going over to do vex.

[Andrew Lawrence]

Once again, I'd like to kindly remind people the problem is most likely not the NI C-RIO. The C-RIO's job is to process the information given to it from the sensors and inputs from the radio and use that to control the various aspects of the robot.

The problem most likely is the radio. The C-RIO does not connect to the driver station at all. It's the D-link radio that does.

The C-RIO was introduced 4 years ago in 2009 to help in handling the vision processing that was introduced more heavily in 2009 than any other game, and to accommodate for the old IFI technology. The radio, on the other hand, wasn't chosen too well, and as a result left many teams without connectivity to the field.

tl;dr: It's most likely the radio's fault, not the C-RIO's.


EDIT: I am making these accusations because while it certainly is possible it is something else, multiple robots in every match had comms issues. Some of the teams never had comms issues before. I don't have a field diagnosis, but this is my result from troubleshooting the problem with what I have.

[techhelpbb]

A big move needs to be made by FIRST here and if it isn't done I suspect much fallout in the frc community with more teams just going over to do vex.

As a mentor I think that the current control system (the entire control system including the radio) offers the opportunity to teach about troubleshooting. The problem is that while the opportunity to learn troubleshooting appears the tools, access and freedom to do that troubleshooting does not.

When I worked as the spare parts person at MAR Mount Olive I brought my oscilloscope. It surprised me the gulf between the teams that actually used it and the teams that neither knew what it was or even had any idea of what to do with it on a FIRST robot. The difference in levels of understanding was not between the veteran teams and the new teams were you'd think it would be.

I think that FIRST needs to very carefully reconsider their process for troubleshooting. Otherwise I suspect that the only thing a highly integrated control system does is remove students from the equation in favor of placing the responsibility directly on FIRST.

I respect either aspect of the matter. I'd just hate to make FRC operate on the equivalent of the LEGO NXT system (and I'm not knocking LEGO NXT I have been an FLL judge for years).

[ticoloco12]

Hello all,

Now that the dust has settled, if only a tiny bit, I'll share what I saw from Einstein. You may know me as Will, I work with team 207's strategy and drive team. I am a FIRST Alumni and have been mentoring for 4 years now. let me make it clear that Team 207 is home and nothing but thankful to team 233 and 987 for an amazing time at world champs and our first trip to Einstien. Through the chaos that was the Finals, they showed that they are pure class and of the highest examples FIRST has. (341, 254,and 78 as well)

Yes, many, an unforgivable number in fact, of robots were either dead or experienced significant periods of losing coms. Some, clearly game changing. This sucked the emotion and prestige of the finals, clear out of every team on the field. Not one of the drive teams, or pit crews in our alliance cheered as we made it into the finals after semi's. It was a feeling of disbelief, that FIRST was allowing such a thing. It would have taken 30 minutes, to replay all of Einstein back to back in my mind. but that did not happen.

As a programming mentor for Pink said earlier, this is not how ANY FIRST team wanted to win or lose. I remember being on the drive team myself and dreaming of one day coming back to mentor my team there, to feel that rush, to cheer to the absolute maximum the human lungs can. To have been dead silent, and not even be able to rally our team between F1 and F2 because everyone's sentiment was, whoever works longer will win, is the largest disappointment I have ever had in my FIRST life. Even if making it there after an EPIC curie division Finals. From what I see in finals videos and scores, 341's alliance in the division finals was as competitive if not more so than any alliance on Einstein.

However, the most important feeling and experience to get away from this, is the spirit of FIRST teams. They are completely separate from fields and external chaos such as sever weather, curses, coms issues and all else. Every team on Einstein was united in their dislike of what happened, and treated each other graciously and professionally. When 987 needed time to inspect their robot and wanted to call a timeout (and I still don't know why we were denied one) 180 came to offer any assistance and tried to call theirs as well. both finalist alliances congratulated each other, on the field, and in the pits. The students were proud and even 118, and the Canadian teams did not hesitate in spreading hugs and handshakes to us all. The students themselves began making plans to replay all matches at IRI with robots in the same condition as they left the field Saturday. Dean Kamen was on the field, as was Dave and the look on their face said it all. They were disgusted and Dean was almost in tears as a Girl from a team I cannot remember pleaded the cause of her team. Dave was gone it seemed by the time the finals were over, and Woodie's face from the stage looked like he was viewing somebody drown and could not help. I insist, the truest qualities of FIRST were never seen in such bright colors and they were here. Technology failed us and perhaps the field crew have a huge catastrophe to correct. These finals will be remembered as the day the field chose the winner. BUT IN NO WAY, SHOULD THESE EVENTS TARNISH THE IMAGE OF FIRST, ITS MISSION, ITS STUDENTS, TEAMS, OR FOUNDERS.

Some of the greatest Robotics matches I have ever seen took place on Curie field in the finals. And I speak for every member of team 207 when I say we are beyond proud of all we did, and all our alliance partners did. We would put money on our alliance anyday, and if we lost a fair match, we'd cheer so loud for the alliance that beat us that our lungs would explode. Regardless of what trophies or banners the fields gave us, the embraces, laughs, comments, and connections with amazing FIRST-ers are sooooooo much more valuable. I truly hope the Einstein replay at IRI happens, not to play these matches, as much as to share them with such amazing teams and individuals, like we should have been able to.

I hope a lot changes next year, but only as far as the field goes with technology, and in organization and error management. the Human aspect of FIRST and it's competitors is something we can be prouder today of than ever.

-Will

[Deetman]

Once again, I'd like to kindly remind people the problem is not the NI C-RIO. The C-RIO's job is to process the information given to it from the sensors and inputs from the radio and use that to control the various aspects of the robot.

The problem is the radio. The C-RIO does not connect to the driver station at all. It's the D-link radio.

The C-RIO was introduced 4 years ago in 2009 to help in handling the vision processing that was introduced more heavily in 2009 than any other game, and to accommodate for the old IFI technology. The radio, on the other hand, wasn't chosen too well, and as a result left many teams without connectivity to the field.

tl;dr: It's the radio's fault, not the C-RIO.

Be careful making blanket assumptions and statements like that without any evidence supporting it. This is really a general statement... we can all speculate as much as we want but I am not aware of anyone (possibly even FIRST) that has hard data that points to one component or another. Anecdotal evidence seems to be pointing many in one way, but there are multitudes of layers to our complex control system in which any one of them could be the cause.

[jblay]

Once again, I'd like to kindly remind people the problem is not the NI C-RIO. The C-RIO's job is to process the information given to it from the sensors and inputs from the radio and use that to control the various aspects of the robot.

The problem is the radio. The C-RIO does not connect to the driver station at all. It's the D-link radio.

The C-RIO was introduced 4 years ago in 2009 to help in handling the vision processing that was introduced more heavily in 2009 than any other game, and to accommodate for the old IFI technology. The radio, on the other hand, wasn't chosen too well, and as a result left many teams without connectivity to the field.

tl;dr: It's the radio's fault, not the C-RIO.
If it is so simple then why didn't FIRST make a change in radios after the immediate issues of week 1?

[techhelpbb]

If it is so simple then why didn't FIRST make a change in radios after the immediate issues of week 1?

It's not that simple.

Some robots have power issues.
Some robots have radio issues.
Some robots have code in the cRIO that causes communications problems.
Some robots have bad components.
Some robots have bad wiring (that sometimes doesn't show up unless it's under load or getting impacted just the right way).
Some robots have issues with video swamping.

I can say that I personally have used the NI cRIO 8 module PLC in industrial use and it's a nice product. A little expensive but nice.

This said there are many, many examples of the D-Link AP's power connector being picky and a possible source of issues. That could be fixed by soldering wires for power to the PCB inside the D-Link (if only FIRST would allow it or provide some like that at each event).

It's not fair to say that we have sufficient quantifiable evidence to lay blame on any one piece of hardware. Good troubleshooting dictates that quantifiable evidence must exist for this to be more than speculation.

[216Robochick288]

I honestly hesitate to answer. Maybe switching fields just simply wasnt an answer. It is hard to say for sure, but I anxiously await what FIRST might have to say about this. I certainly would hope that this gets solved for next year. Still, congrats to The Bomb Squad, Raider Robotics and SPAM.


Also one question that arose on the ride back from a few people: Why would the teams agree to play on a broken field?

In a responce ot this, what were they supposed to do? I see little that the teams could do.

[Ether]

Hello all...

Thanks for sharing your thoughts. It's better to post a link (http://www.chiefdelphi.com/forums/showpost.php?p=1164029) instead of re-posting the exact same text to multiple threads (especially for lengthy messages).

[Tim Reddersen]

"This is where gracious professionalism comes into play." Quote from my son Daniel, one of the drive team members of 2194 Fondy Fire. I have no doubt that the FIRST organization will handle this well. I saw Dean Kamen squatting by the field, looking at things, with concern written all over his face. And I am very gratified to see that the posts here on Chief Delphi are so supportive. You are all awesome and we are still happy beyond words to have been the first Wisconsin team to make it to Einstein. :D Our most sincere thanks to our Alliance members 548 Robostang and 118 Robonauts.

[Johnny_5]

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.


I own a theater company and I manage a few theaters out of my local hometown and use a very similar setup to what they use in the rigging on Einstein. Wireless DMX is very behind in the amount of channels it can support and the lights that move, the intelligent lighting takes up many channels as every single aspect takes one channel. It would not have made any sense for them to have used wireless DMX. Using W-DMX would also mean flying all of the dimmer packs for the lighting, then the rigging isn't balanced on one side if those packs are up there.

I have used 2.4GHz and 5Ghz for intensive applications such as media streaming in theaters that were using wireless DMX at the same time and experienced no interference.

[koolbob23]

I own a theater company and I manage a few theaters out of my local hometown and use a very similar setup to what they use in the rigging on Einstein. Wireless DMX is very behind in the amount of channels it can support and the lights that move, the intelligent lighting takes up many channels as every single aspect takes one channel. It would not have made any sense for them to have used wireless DMX. Using W-DMX would also mean flying all of the dimmer packs for the lighting, then the rigging isn't balanced on one side if those packs are up there.

I have used 2.4GHz and 5Ghz for intensive applications such as media streaming in theaters that were using wireless DMX at the same time and experienced no interference.
Agreed. Using wireless DMX would be way too much work for this kind of setup as well as I thought I saw some DMX wires on the metal bridges they were using to hold everything up. Getting all the lights to work correctly is enough work without factoring in wireless well at least for me.

[Duke461]

How exactly would we go about rebooting our crio mid match when we have no communication with our robots? Walk out on the field and cycle the main breaker?

We will share our full thoughts and experiences in the coming days.

Well, I'm in no way an electronics guy, so I may not have worded this correctly. However, there's something you can do on/through your driver station to do something similar to "restarting the cRio/radio/something". We had no comm's or code and we clicked something on our Driver Station during the match to fix it (I was not the one doing this so I'm not 100% sure what they did). If someone knows what I'm talking about and could explain what I mean further, that would be awesome.

-Duke

[basicxman]

Well, I'm in no way an electronics guy, so I may not have worded this correctly. However, there's something you can do on/through your driver station to do something similar to "restarting the cRio/radio/something". We had no comm's or code and we clicked something on our Driver Station during the match to fix it (I was not the one doing this so I'm not 100% sure what they did). If someone knows what I'm talking about and could explain what I mean further, that would be awesome.

-Duke

There is a button to soft restart a connected robot's cRio. What you're describing likely didn't affect the cRio.

[robostangs548]

I have been a student and FIRST mentor since I was 8 years old. I am now 21, and with school and work now consuming my life, I will be announcing my resignation from the team that I have grown to love and hold as my number one priority over the last few years. I have grown with this team from the "new-beginning" where we lost all sponsorship my freshman year, and paid mostly out of pocket to build and compete in the FIRST Robotics Competition.

Since this was what I considered the "last year" I decided to dedicate most of my time, mentoring and helping build both an amazing team, and robot. Along with the a dedicated team, thousands of hours, and a hard earned budget, this was the first experience I have had on Einstein. I can remember watching all of the great teams that I grew up with on Einstein from 1998-2011, and dreaming that with enough dedication, fund-raising, and overall team effort, we would one day be able to compete with those teams. Our team worked to find sponsors, mentors, and build a dream robot. After many all-nighters and extreme dedication, finally, that opportunity came with a near perfect alliance of 118, 2194, and 548.

Though it is still unknown what the root cause of these issues were, I am amazed that with a overall financial contribution as large as those in which large corporations and other donors have contributed, along with the number of engineers and other volunteers involved, FIRST was still unable to fix an issue that was known week one.

My "pit-crew", along with others, sat field-side disheartened by the fact that technology was failing not only our alliance partners, but also all of our opponents. An image that sits burned into my brain is the "This is FIRST" video that used to be on the FIRST website, where one of the mentors says "Months of Work.... Sat there.... Dead...." and really brings an emotional side out of me that can't be put into words. I can say with a large level of confidence, that this issue had nothing to do with the robots, but rather the field. I think the scores of these Einstein matches were a clear indicator of how messed up the outcome of this really was.

A team like 118, who is mentored by engineers that work on projects that literally leave our planet, who had a robot that not once had an issue through our entire Newton competition, without even a slight modification in code, were handicapped with a robot that "Sat there.... Dead...." every match on Einstein with the only reason being that of field coms.

118 proceeded to be lectured by an FTA that it definitely was not field issues, but rather issues with their robot. They pulled up a report from their robot only to see that the field could definitely be the reason for the loss of communication at the start of the match. The FTA said it could be their cRio having an issue, so they swapped out to a new cRio only to have results matching those from matches prior.

I was amazed how teams on the field pulled together, worked as a team, and almost every team said "we did not want to win this way." All teams were upset at one point or another with the outcome of these Einstein matches. From the field, we heard the lack of cheering, and could see the look of utter shock in the faces of everyone in the audience. Though the FTA's did work hard to help solve this issue, I believe that FIRST had no idea what was going on. Do I think they made the right decisions in how they handled the situation? Far from it. Do I think they knew what the issue was? Absolutely not. But after that entire disheartening fiasco, I would assume they wont let it happen again. I think that all the teams were amazing, and though it was upsetting, it was an honor to leave FIRST surrounded by everyone I grew up admiring.

Congrats 16, 26, 180.

[Grim Tuesday]

Our most sincere thanks to our Alliance members 548 Robostang and 118 Robonauts. :cool:

Congratulations from one of the Newton finalists!

I am in a minority that think the issues were handled correctly, and in accordance with current FIRST policies. After the first two matches, they did exactly the right thing (and somewhat outside precedent) with replaying them. The next matches, there were scattered problems, but in accordance with what is de-facto policy, they did not replay or cancel them. They did not move to Galileo because it was already torn down. They did not crown twelve teams Champions, because to do that would be to admit defeat completely and utterly. FIRST never does that. It appeared that the issues were fixed when 4334 began working.

Sidenote: We were one of the backup bots for the Newton Division, so our drivers were back behind Einstein if anyone has an questions about what was actually going on. We all know the saying about assumptions.

In the first finals match, it didn't appear from my position in the pit next to Einstein by the VIP section that there were any field issues. It is clear that there were in the second match, but we have to think about the enormous pressure here. Was the decision deferred to the head ref? Did Dean Kamen make the call himself? Everyone is talking about how rustled they looked, and I can attest, Woody looked somewhat put out when he was reading the awards. Dean looked nervous and taut while standing next to the field. There is a human element here, and the stress and pressure of deciding a world champion is enormous.

Regardless, if the FTAs could detect no problem with the field, then they made the correct call in not replaying the match. It follows all precedent. I covered the reasoning in not declaring all teams World Champions above. That isn't to say that the field issues on Einstein were disgraceful.

The issue was not with how the problems were handled. They were with the field/control system. It means that there are quite a few things that need to be greatly looked at, and fixed before next year:

There have been many high profile cases of very high tier teams being unable to connect to the field (118 and 1676 among others). This needs to be fixed by next year, especially the fact that some fields were able to be connected to and others not.
Einstein needs to be tested. It is supposed to be the best field in all the land, and if there are problems on it, it becomes a massive debacle (ie, this thread). Someone suggested running a practice match on it with divisional quarterfinalists first. This would diminish the 'virginity' of the field, as well as reduce the value of "being on einstein". I suggest that the GDC brings the 6 bots they build to test the game when they're making it and run the tests that way, being careful not to scuff anything. They can do it quietly whenever they please, maybe Wednesday night.
The idea of Einstein needs to be reexamined. It seems somewhat stupid to put all your hopes on an untested field. Why not play the finals match on the field of whichever division won last year?


I think we need to thank all the Einstein teams for remaining Gracious and Professional, role models for the whole community.

[Katie_UPS]

The idea of Einstein needs to be reexamined. It seems somewhat stupid to put all your hopes on an untested field. Why not play the finals match on the field of whichever division won last year?


My friend and I examined solutions on the 6 hour ride home:

-Putting the finals on an existing field poses the logistical problem of seats. The teams on Archimedes (if that were the chose field) would have the advantage over every other division when it comes to getting good seats for "Einstein". Also, how would FIRST block off seats for the Einstein teams? You can't block those out for all of Saturday if they're in the middle of a division's block of seats, and you can't really kick teams out of those seats in time for "Einstein".

-Moving fields during awards (after the issue was noted) is a logistical nightmare. The podium, cannons, judges seats, student rep seats, etc are all set up to be where they are, and you can't just move that in a quick and organized fashion. Not to mention stampedes of people trying to move seats.

Anything else we've said has been mentioned and scrutinized already.

[sgreco]

-Moving fields during awards (after the issue was noted) is a logistical nightmare. The podium, cannons, judges seats, student rep seats, etc are all set up to be where they are, and you can't just move that in a quick and organized fashion.

I see your point, but nobody wanted to see what was going on anyway with robots not moving, and basically illegitimate eliminations on Einstein. They could have finished the awards while the teams move, then transfer to Galileo once they didn't need the podium anymore. At this point, who cares if every spectator was able to get a good seat? It's better to have legitimate elimination rounds then to have everybody disappointed watching illegitimate ones.

Also, I'm not sure how the screens worked, but it was probably reasonably easy to get a feed from another field displayed on the screen at Einstein. That way, not everyone has to move.

[Gary Dillard]

I wasn’t sure which thread to post this in, but I figured this was the most appropriate. For those who don’t know, I was a mentor with SPAM from their rookie season until 2006 and I was on stage in Einstein with them this weekend. Three of the current mentors on the team were students on the team when I was there, they have returned after college as professionals, and are shining examples of the best that FIRST has to offer. I am not going to say that I speak directly for them or any of our alliance, but I’d like to share some of my thoughts and emotions.
First, I think our sentiment is best summed up by the way Karthik and I greeted each other on stage at the end of the event. I said “We’re having kind of a tempered celebration” and he replied “I understand”. We would have replayed matches all night if necessary; when we were told we were replaying match 1, no one even batted an eye. We didn’t want to win this way, but it is what it is. We consider ourselves World Champions; we felt like we had an unbeatable alliance, but I’m sure the other division winners felt the same way. Although our celebration was somewhat muted, our loss of elation pales in comparison to what I’m sure the other division winners are feeling about what happened. After we lost the championship finals in the third match in 2002, I replayed that match over and over again in my head –if we had only done this or that we might have won. What are the Robonauts or the Pink team going to imagine that they might have done differently? My heart goes out to you.
As far as the OP’s question, I’d like to mention part of what the head ref Dr. Aidan Brown told the alliance captains before the final four started (as relayed to me by our team captain, so I’m not quoting and I don’t remember the exact words). He said this is a competition, but it is also a show. (He also told them to have fun). I fully understand this, and the show must go on, so some decisions for the benefit of FIRST and the ceremony had to be made at the expense of the competition. I think it was a lose – lose situation, and this was the best they could do. This could have been prevented long before Einstein, but that is a discussion for other threads which we have all been following all season. I saw one FTA in tears afterwards, and I know everyone involved was doing all they could. 10 years ago when we were on Einstein, it wasn’t part of the closing ceremonies; we just played in front of whoever was still there in the afternoon, and we could have replayed without much fanfare, and then the closing ceremonies were in the evening. When they switched to the current format, I thought it was fabulous and wished we could have had that experience in ’02. I still feel that way, so I guess this is the compromise we have to accept. Thankfully, a lot of the corporate leaders were at the division eliminations matches up close and got to experience the thrill of the competition, and hopefully their support will continue so that we can advance the mission of FIRST.
There were some discussions amongst the teams on stage about ways to make a public demonstration, including the winners declining to accept the championship and having all 12 teams go up as finalists. This discussion took place after the semi-finals and before the finals, so we didn’t know yet that we would be the winners (in fact, we knew we would be the red alliance and we were worried). I went back and forth on it, weighing the need for a statement versus the ramifications to FIRST. Our alliance wasn’t unanimous in what to do, so we decided to just accept the results and proceed as normal.
As far as having the teams go to IRI and play it off (and I haven’t talked to anyone on SPAM about this, but it’s just my 2 cents), if you’d like to write the $10,000 check to SPAM for them to travel, and cover the mentor’s lost wages for another 2-3 days missed at work, maybe they’ll be able to get the paperwork through the school system in the month they have remaining to allow an out of state / after school is out trip to take place. I’ve only been able to make to IRI once, when my daughter wanted to visit Purdue as a possible college and we made the trip coincide with IRI. The rest of the time I have a real job and a family that I like to spend time with and work to do around the house to catch up on from the 2-3 months I neglect it during build and competition season. Not that the team wouldn’t love to attend IRI (for all I know they already had it in this year’s plan and budget), but it doesn’t just happen on the spur of the moment.
Finally, thank you for the respectful and courteous way that this discussion has taken place; pretty much everyone has caveated their comments with “not taking away anything from 16,25,180”, etc., and that means a lot. FIRST is a big family and we‘ll get through this together.

[Katie_UPS]

I see your point, but nobody wanted to see what was going on anyway with robots not moving, and basically illegitimate eliminations on Einstein. They could have finished the awards while the teams move, then transfer to Galileo once they didn't need the podium anymore. At this point, who cares if every spectator was able to get a good seat? It's better to have legitimate elimination rounds then to have everybody disappointed watching illegitimate ones.

Also, I'm not sure how the screens worked, but it was probably reasonably easy to get a feed from another field displayed on the screen at Einstein. That way, not everyone has to move.

I don't want to get in a back and forth, so this will be my last post (and we can PM if you'd like to continue conversation).

Finishing awards while teams move: drive teams? teams in the stands? That's pretty disrespectful to the teams winning awards and the speakers presenting them. No one feels too great about themselves when they're talking/being presented and the audience is too preoccupied with something else.

And teams not moving but the matches moving? You can't stop people from moving and I'm pretty certain 80% of the FIRSTers I know would want to move to actually see the robots.

I'm not saying I'm glad things turned out the way they did. I couldn't quite get excited about Einstein after the issues weren't fixed. I'm just saying that FIRST didn't have too many options and had legitimate reasons for not switching fields.

However, I wouldn't mind them presenting all the awards while field crews and FTAs switched all the FMS components of the fields (have one crew gut Einstein's FMS and another gut the most reliable field and then swap). I would gladly take a delay of any time span to fix the problem (and I'm sure you're thinking "well then we can just switch fields", but that's a hairier mess, in my opinion, than switching field elements).

[sgreco]

I don't want to get in a back and forth, so this will be my last post (and we can PM if you'd like to continue conversation).

Finishing awards while teams move: drive teams? teams in the stands? That's pretty disrespectful to the teams winning awards and the speakers presenting them. No one feels too great about themselves when they're talking/being presented and the audience is too preoccupied with something else.

And teams not moving but the matches moving? You can't stop people from moving and I'm pretty certain 80% of the FIRSTers I know would want to move to actually see the robots.

I'm not saying I'm glad things turned out the way they did. I couldn't quite get excited about Einstein after the issues weren't fixed. I'm just saying that FIRST didn't have too many options and had legitimate reasons for not switching fields.

However, I wouldn't mind them presenting all the awards while field crews and FTAs switched all the FMS components of the fields (have one crew gut Einstein's FMS and another gut the most reliable field and then swap). I would gladly take a delay of any time span to fix the problem (and I'm sure you're thinking "well then we can just switch fields", but that's a hairier mess, in my opinion, than switching field elements).

This'll be my last post about it too. I meant just move the drive teams, robots, refs, field crew etc and keep the crowd where they are until awards are given out. Then when they're done with the podium and the presentation, let people move. It would present some logistical issues with everyone moving at once, but I feel it would have been better than having illegitimate matches played on Einstein. People have enough class that if you told them to wait before they move, they'll wait. Then you put it on the jumbo-tron in case not everyone got a seat at the new location.

[Kit Gerhart]

The field issues seemed to be just with the red alliance..

We were on the blue side and died for an extended period of time in one match on Einstein, and briefly in others. At least one of our partners died for a time, and had glitches. We had no such problems on the Curie field. Regarding who "should have won," we'll never know for sure, but if played on a properly working field, it is quite possible that the Canadians would have beaten us, and we never would have made it to the finals.

As far as solutions, this year is done, but for next year....

1) One of the division fields could be used as Einstein for the "superfinals." It would have been well-tested after running over 150 matches. To my knowledge, we had no field-related issues on the Curie field during 3 days of play. If i'm wrong on that, please correct me.

2) Teams could play most of their matches on their "home" field, but be rotated through the Einstein field as was done at Disney, and maybe since then. That way, the field would be well tested, and presumably fixed, if there were problems.

[Azores]

"Dear FRC Teams:

Thank you for your incredible enthusiasm and Gracious Professionalism throughout the year and at the Championship.

We apologize for the technical problems that affected the final matches at our Championship. We will examine all of the facts, report our findings and ultimately solve any and all identified issues.

Sincerely,
Jon Dudas
President, FIRST "

Just received this, posting it here in case anyone doesn't receive it.

[techhelpbb]

Until we know exactly what the problem(s) are, everything in the entire control system (cRIO, Radio, DS, FMS, etc) should be considered suspect and investigated.

I agree.

Work through the OSI model on both ends of the link.

http://en.wikipedia.org/wiki/OSI_model

It's complicated...but by process of elimination I'm sure the problems can be found now and in future developments.

[efoote868]

2) Teams could play most of their matches on their "home" field, but be rotated through the Einstein field as was done at Disney, and maybe since then. That way, the field would be well tested, and presumably fixed, if there were problems.

As much as everyone would like Einstein to be in "pristine" condition, I think this is a good solution moving forward. It doesn't change the format of play at the end, and it has a very good chance of working out field issues (and if the FMS needs to be updated/replaced, we'll have ample warning).

[artK]

For those complaining that Einstein's prettiness would be lost in test matching before finals, They could use used electronics and virgin game components. They are separable, Our team uses the game components without all the electronics (like score counters and displays from the walls of the station). It's not like you can tell whether electronics have been used before.

[thefro526]

I believe I may have a relatively Simple Solution to prevent such an issue from happening again* - well actually two ideas on the same train of thought.

What if we were to Give each Alliance an Introduction Match on Einstein? When an Alliance is sent to Einstein, they make it a point to introduce them either via Video or by having them walk onto the field. Why not give each alliance the field for 2 minutes and 15 seconds and tell them 'Show 'em what you've got.' This serves a variety of purposes:

-Each Alliance Has what's essentially an uninterrupted Practice Match on Einstein. They can check Camera Calibration, Code, Tweaks, Communications, etc. If each robot runs during this intro match, you've now ruled out a bunch of machine issues from causing a robot to sit dead.

-These intro matches are going to be AWESOME for spectators. Imagine 2056, 1114 and 4334 on the field, uninterrupted driving the score up. It would be an EPIC sight. This gives everyone watching a good chance to get a feel for who's going to be playing on Einstein.

Or, Why not play Division Finals On Einstein? Basically, we'd be turning Einstein into a Full Elims Bracket with the Division Finals now serving as the Einstein Quarters.



1) One of the division fields could be used as Einstein for the "superfinals." It would have been well-tested after running over 150 matches. To my knowledge, we had no field-related issues on the Curie field during 3 days of play. If i'm wrong on that, please correct me.



Kit, I believe you are correct. As far as I am aware, the only team that had multiple issues with their robot not running on the field was 971, and the issue was on their end IIRC.

78's Dying in Final 1 was due to the radio coming unplugged.

*The real solution to all of this is to make a 100% reliable field, but that's been covered enough here.

[Sean Raia]

I'm a bit worried that many people are writing this off as an "oh its cause they didn't test Einstein" issue. Really? Did you hear about 1717's issues in their division?

The problem here is deeper. These issues persist on many FRC fields and this needs to change.
Ultimately the firmware and hardware we are given by FIRST needs to be built tougher and respond more reliably.

Everyone contributes far too much money and time to be met with faulty reliability on FIRSTs end.

[Anupam Goli]

I believe I may have a relatively Simple Solution to prevent such an issue from happening again* - well actually two ideas on the same train of thought.

What if we were to Give each Alliance an Introduction Match on Einstein? When an Alliance is sent to Einstein, they make it a point to introduce them either via Video or by having them walk onto the field. Why not give each alliance the field for 2 minutes and 15 seconds and tell them 'Show 'em what you've got.' This serves a variety of purposes:

-Each Alliance Has what's essentially an uninterrupted Practice Match on Einstein. They can check Camera Calibration, Code, Tweaks, Communications, etc. If each robot runs during this intro match, you've now ruled out a bunch of machine issues from causing a robot to sit dead.

-These intro matches are going to be AWESOME for spectators. Imagine 2056, 1114 and 4334 on the field, uninterrupted driving the score up. It would be an EPIC sight. This gives everyone watching a good chance to get a feel for who's going to be playing on Einstein.

Or, Why not play Division Finals On Einstein? Basically, we'd be turning Einstein into a Full Elims Bracket with the Division Finals now serving as the Einstein Quarters.



[/I]
I really like this idea. The full bracket would take some time, but the practice matches would be good pumping for the crowds and I would love to see all alliances and what they are capable of before the matches start.

[techhelpbb]

I'm a bit worried that many people are writing this off as a "oh its cause they didn't test Einstein" issue. Really? Did you hear about 1717's issues?

The problem here is deeper. These issues persist on many FRC fields and this needs to change.
Ultimately the firmware and hardware we are given by FIRST needs to be built tougher and respond more reliably.

Everyone contributes far too much money and time to be met with faulty reliability on FIRSTs end.

You are correct. The issues on Einstein were just more visible. Issues have been ongoing all season at multiple venues.

I'm not entirely sure that all the problems are firmware and hardware provided by FIRST. In some cases teams did have some influence over the issues. Not that teams didn't expend absolutely every effort to work out their issues when they had the tools to find them.

However, I am entirely sure that FIRST needs to work on the troubleshooting process. Not merely for Einstein but competition wide. FIRST needs tools to solve these problems or no matter the brilliance of everyone involved the deadlines will be a disaster.

[CalTran]

What if we were to Give each Alliance an Introduction Match on Einstein? Why not give each alliance the field for 2 minutes and 15 seconds and tell them 'Show 'em what you've got.' This serves a variety of purposes:

-Each Alliance Has what's essentially an uninterrupted Practice Match on Einstein. They can check Camera Calibration, Code, Tweaks, Communications, etc. If each robot runs during this intro match, you've now ruled out a bunch of machine issues from causing a robot to sit dead.
-These intro matches are going to be AWESOME for spectators. Imagine 2056, 1114 and 4334 on the field, uninterrupted driving the score up. It would be an EPIC sight. This gives everyone watching a good chance to get a feel for who's going to be playing on Einstein.

So like a complete exhibition match where teams can go in guns blazing? I would LOVE to watch that. Maybe even throw in a 2v2 of all of the #1 seeds. ACvs.GN

[sgreco]

I'm a bit worried that many people are writing this off as a "oh its cause they didn't test Einstein" issue. Really? Did you hear about 1717's issues?

The problem here is deeper. These issues persist on many FRC fields and this needs to change.
Ultimately the firmware and hardware we are given by FIRST needs to be built tougher and respond more reliably.

Everyone contributes far too much money and time to be met with faulty reliability on FIRSTs end.

This.

100% correct. These problems were occurring on other fields, at other competitions, and other years with this control system. Something needs to be done about the system, not the implementation. The system they have is the problem, not specifically how they went about handling Einstein (though that could have been improved too, it's not the root of the problem).

[Nuttyman54]

I would like to start off by congratulating 180, 25 and 16 on their win and performance, it was an absolutely awesome alliance to behold.

Regarding the handling of Einstein...I felt like there may have been a better way to handle it, but I don't have a solution for how to do it better, other than to give more time to the field crew by running awards consecutively, and possibly not to crown a champion. I do understand, however, that there were very limited options. I understand it's supposed to be a "show", but honestly, proceeding with a tournament which is obviously flawed is a pretty bad show. Ultimately, however, there was no good solution. Everyone lost on Einstein because it was impossible to play fair matches. I hold the utmost respect and trust for the veteran crew that was running Einstein, and I believe they did everything in their power to fix it, but at that point things were moving too fast and the problem was too far out of their hands to reasonably do anything else. The decision was made to move on, and I know everyone on the field at the time understood the implications of that decision.

The problem with admitting that there is a serious issue that cannot be resolved is that it throws doubt on the entire season. These problems have been happening since week 1, and to admit on Einstein that the issues were not, in fact, able to be fixed by teams means that everyone who experience this issue at some point during the season would feel cheated.

The problem with proceeding with play is that people will forever question the legitimacy. There is no doubt in my mind that all 4 of the alliances on Einstein had a legitimate shot at the championship, and they all exhibited phenomenal play throughout their divisions. While the outcome may have been the same, I think everyone including the teams on the field will agree that Einstein was decided by the system issues, not by the alliances' play, and that is a real shame.

Ultimately, the solution is for FIRST not to ignore the problems like this when they first start cropping up, and make a concentrated effort to diagnose and resolve them as soon as possible, instead of proceeding with the status quo that anything which can't be diagnosed from the field must be a robot problem. This was not an isolated incident, it was a widespread problem. Enough so that I predicted on Thursday (and several people can quote me as such) that every single finals matchup played would have a dead robot during at least part of it. I don't know if it came entirely true, but the fact that I was right for Archimedes, Curie (I heard Newton had a dead 68 as well, but I didn't personally witness it) and Einstein means that something is very very wrong. That doesn't even count other dead robots during quarters and semis like 330 and 1717.

I would never EVER wish for this to happen to a team, but at the very least, FIRST is now recognizing that they need to take a much closer look at the system. I expect this problem will not happen next year.

The last point I want to make is similar to what Kevin Sevcik said earlier. This may or may not be a "field" problem, and it may or may not be a "robot" problem, but it is very definitely and obviously something which many many top teams and engineers have not been able to diagnose and do not have the ability to fix. There is a problem with the control system and it is something the teams do not currently have any way to prevent from happening, because nobody can tell them how to fix it. This, to me, is inexcusable. I don't care if FMS says everything is working fine, if the best teams in FRC are having that many problems and NOBODY can tell them why, you can't just tell the teams "tough, it's not our end so it has to be yours". I mean no slight to the FTA's, they were just following protocol and doing their best job to diagnose things on the fly. I mean in the sense that FIRST has not publicly expressed any concern or attempt to resolve the problems. If they had been, I really wish they had been public about it.

I know NI has been looking at reports of these issues throughout the season and trying to root out the problem, but I haven't heard what their theories are and if there's any known solution. Likewise, I don't know if their investigation was prompted internally after seeing it at events, or by FIRST.

TL;DR, FIRST needed to be more transparent and proactive about addressing the issues throughout the season. It was apparent to many people that there was a serious problem, but by the time it surfaced on Einstein, there was no good solution.


1) One of the division fields could be used as Einstein for the "superfinals." It would have been well-tested after running over 150 matches. To my knowledge, we had no field-related issues on the Curie field during 3 days of play. If i'm wrong on that, please correct me.


Kit, I believe you are correct. As far as I am aware, the only team that had multiple issues with their robot not running on the field was 971, and the issue was on their end IIRC.

971 experience the same symptoms as exhibited on Einstein during several of our matches on Curie, and we were unable to diagnose the issue, just like everyone else. I'm not particularly familiar with the control system so I can't share many details, but we never determined the root cause of the problem. It may or may not be the same as what other teams experienced. Saying the issues was "on our end" is the same thing teams have been told all season. "Our end" doesn't mean "our fault" or that we had the ability to fix it. We worked in our quarterfinals matches, but we don't know why.

78 also died in F1, but I can't speak to the cause either.

[mcdura207]

Okay, somethings wrong here. In the last 4+ hours since championships have been over, there have been 14 posts in the 'congrats 16, 26 and 180' thread. There have been hundreds complaining about the field issues. Whether you like it or not, they are the FIRST World Champions. It isnt their faults that there was a problem with the field. Their achievements are just as important as field issues. I suggest that everyone heads over to the thread to congratulate them.

http://www.chiefdelphi.com/forums/showthread.php?threadid=106040

and you know if you compare the teams side by side with stats and hard evidence it is apparent who the real world champs should be.

[dsherw00d]

i mentioned in the other thread. This is a mission critical network during the competition. If I was in charge and my network engineering group came to me with failures and could not tell me why, it would be unacceptable. They need the tools and knowledge to track every statistic and frame/packet on the network. Like most engineering groups, they are probably underfunded, understaffed, and lacking tools to adequalty manage the network. Any dropped/delayed frame/packet should throw up a red alarm on a management system in under a second. All traffic should be mirrored to a 2nd port where it is saved. Using this saved file, an entire match can be run offline using a tool such as Wireshark. We also need a way to monitor traffic on the robot's bridge. If we know the root cause, we can fix it. There will always be exceptions which will have to be dealt with, but having he same thing take place more then a couple of times is avoidable. It may take a more exspensive device, but we need more visibility into the communication between devices. After this year, I think more teams will be debugging and monitoring during testing so they know what is "normal". It's another area that will benefit the students to learn. Everyone grab Wireshark and connect a PC with it to the DLINK and start playing:)

Congrats to the world champs!! Fantastic job!!

[James Kuszmaul]

971 experience the same symptoms as exhibited on Einstein during several of our matches on Curie, and we were unable to diagnose the issue, just like everyone else. I'm not particularly familiar with the control system so I can't share many details, but we never determined the root cause of the problem. It may or may not be the same as what other teams experienced. Saying the issues was "on our end" is the same thing teams have been told all season. "Our end" doesn't mean "our fault" or that we had the ability to fix it. We worked in our quarterfinals matches, but we don't know why.

Just to add a bit more information to the mix:
Although we certainly ceased to function in ways similar to the robots on Einstein during a couple of our matches, at least two of the matches in which we ceased to function were certainly caused by problems at our end (In Qualification 9, the joysticks didn't work, and in one other match, the main breaker tripped. We aren't entirely sure as to the cause of these problems, but they did not recur and they certainly weren't the fault of the field). As for all the other times we didn't function, we aren't sure why, but either it was the same as Einstein or it was a different problem. If it is the same problem as whatever happened on Einstein, then we know that we aren't alone in having the problem. If it is a problem with something on the robot, then that means that we may be able to debug it with our own resources. Just so long as FIRST is trying to find a fix to the problem, then I am satisfied. And given the email sent by FIRST earlier, it appears that I am satisfied :).

[nuttle]

Just in case people who are looking into the issue haven't already considered doing this, I think one thing I would certainly do is to set up 'tcpdump' to capture an entire match where the problem reproduces. In order to reproduce the problem, I'd consider using 'netem' to simulate an impaired network. What you really want to see happen at all times, but particularly under adverse conditions, is that each team gets an equal fraction of the available bandwidth, with the most important packets getting through and things like the video stream being throttled. It isn't too hard to make this sort of thing happen using TCP/IP. What you really don't want to happen is for any team to lose enough bandwidth that heartbeats get held off for long enough to cause problems.

We had very similar problems in 2010, when rookie teams had a different radio than veteran teams. This seemed to hit Red 2 especially hard, IIRC. I imagine the teams on Einstein all had this covered, but we have always tried to mount the radio in the open, high and not surrounded by metal, as well as put a glob of hot-melt glue so it is holding the power plug in place. If your team doesn't already do these things, they can only help.

Congrats to all, it was hard to watch things unfold as they did but it is a fantastic achievement to make it onto Einstein and no one should let what happened mar the pride in finishing in the top 12 teams in the world. The Chairman's Award was very well deserved and more than anything, FIRST really is a fantastic program that continually works to improve and everyone involved deserves to be recognized for inspiring everyone else and for demonstrating Gracious Professionalism under very intense circumstances.

[dsherw00d]

Nuttle - you are a CLI guy:) Your asking alot - learn UNIX and use a command line. Holy Cow, my heads going to explode:D

Good advice though - you don't have to spend any money to do some really cool stuff.

[Dave McLaughlin]

Perhaps they should have been playing this remix instead of the FIRST Robotics version on Einstein...

http://gobarbra.com/hit/new-4e6fed9c8f317fd0407128c25cdfe22f

My apologies to the teams, this was handled very poorly in my book and it is my opinion that you should all walk away feeling like champions.

[steverk]

I'm a bit worried that many people are writing this off as an "oh its cause they didn't test Einstein" issue. Really? Did you hear about 1717's issues in their division?

The problem here is deeper. These issues persist on many FRC fields and this needs to change.
Ultimately the firmware and hardware we are given by FIRST needs to be built tougher and respond more reliably.


I agree! There were problems all season and I've heard reports of problems on all of the fields at CMP. However, the problems were worse on Einstein.

So what is different about Einstein that might explain things?

A few have been identified. More spectators means more texting, wifi, etc. The field was untested. The spectacle was greater with more lights, etc. The storm rolled in.

One wayt to troubleshoot is to take a list and add anything else that you can think of and then eliminate them one at a time. I'm sure each of us have our favorite theory of what was the cause.

Just to plug for mine, the problem appears to be intermittent, tends to affect teams with lots of comms back and forth to their staion, and tends to happen during elims when the fans are paying attention and using their cell phones and wifi tethering a lot. Therefore, I think it is radio interference from a combination of sources. If that is correct, then this is going to be very hard to pin down concretely, but realtively easy to fix. A different radio or frequency could fix the problem.

[nuttle]

My guess is that the issue is triggered by more bandwidth being needed than there is available. So, robots that upload lots of telemetry, all have video feeds, etc. combined with anything that causes less availible bandwidth is the magic combination. Fixing whaever is limiting the bandwidth makes the problem go away, but really doesn't solve the deeper issue. When bandwidth is tight, packets drop, TPC requests retransmissions, and this only results in more bandwidth being needed. Traffic on some critical connections gets held up for long enough that robots die and do not recover. If this is right, the old radios in 2010 turned up the same underlying issue but the problem went away when everyone was told to use the new radios.

Depending on how things are implemented, a single robot could have more than one video stream coming from a single camera -- one for the dashboard and another one for off-board target tracking. One way to simulate this would be to open extra web sessions with the camera. The 'netem' tool I mentioned before can simulate a network with various issues in the network and is a more controlled way to try to cause this type of problem.

This wouldn't be an issue when using a tether, and a single robot running over wireless would have six times the bandwidth as when a match is being played and so would not likely see this either. The exact traffic when running under the FMS might be different as well. Using tcpdump / wireshark would allow digging into the problem.

Just thought I'd throw this out since this angle hasn't come up in this thread and it is one theory that should be considered, I think. THe solution would involve taking steps to make sure the critical data always gets through by limiting the bandwidth that is allowed for less critical data when bandwidth is tight. Giving each team an equal amount of bandwidth is only fair, and things like the video feed will do OK with limited bandwidth, usually by dropping frames. It would be good to have a way for teams to test with limited bandwidth as well. It would be possible to have a gauge on the driver station showing bandwidth used or something along these lines to help teams catch issues they can contol that cause them to use more bandwidth than they need and generally be more aware of this.

[Jared Russell]

78 also died in F1, but I can't speak to the cause either.

78 died due to the power connection to the wireless bridge coming out after an impact with the barrier shortly after hybrid mode.

[Bruceb]

If you loose comms due to something on your robot unplugging then it is clearly your problem. Been there done that. BUT these field issues are embarrassing to FIRST or at least should be. I am not an engineer but if I remember right we are still on communications with Voyager 1 at something like a bazzillion miles from here and it is using a 5 watt transmitter. There has to be a better more robust more resilient way to make this work even if we have to go to a completely different control system.
I am not pointing fingers. I know it is a difficult situation to debug. It is just so frustrating for all involved. We were smacked by this in 2010 and it is still an issue. Curie had very few if any of these issues. What was so different there?

Bruce

[Racer26]

My guess is that the issue is triggered by more bandwidth being needed than there is available. So, robots that upload lots of telemetry, all have video feeds, etc. combined with anything that causes less availible bandwidth is the magic combination. Fixing whaever is limiting the bandwidth makes the problem go away, but really doesn't solve the deeper issue. When bandwidth is tight, packets drop, TPC requests retransmissions, and this only results in more bandwidth being needed. Traffic on some critical connections gets held up for long enough that robots die and do not recover. If this is right, the old radios in 2010 turned up the same underlying issue but the problem went away when everyone was told to use the new radios.

Depending on how things are implemented, a single robot could have more than one video stream coming from a single camera -- one for the dashboard and another one for off-board target tracking. One way to simulate this would be to open extra web sessions with the camera. The 'netem' tool I mentioned before can simulate a network with various issues in the network and is a more controlled way to try to cause this type of problem.

This wouldn't be an issue when using a tether, and a single robot running over wireless would have six times the bandwidth as when a match is being played and so would not likely see this either. The exact traffic when running under the FMS might be different as well. Using tcpdump / wireshark would allow digging into the problem.

Just thought I'd throw this out since this angle hasn't come up in this thread and it is one theory that should be considered, I think. THe solution would involve taking steps to make sure the critical data always gets through by limiting the bandwidth that is allowed for less critical data when bandwidth is tight. Giving each team an equal amount of bandwidth is only fair, and things like the video feed will do OK with limited bandwidth, usually by dropping frames. It would be good to have a way for teams to test with limited bandwidth as well. It would be possible to have a gauge on the driver station showing bandwidth used or something along these lines to help teams catch issues they can contol that cause them to use more bandwidth than they need and generally be more aware of this.

This concept DOES hold some water, especially in the context of Einstein.

It's reasonable to believe that the robots that reached Einstein would be making more effective use of the bandwidth available (sending streaming video back to the DS, etc).

It's then also reasonable to believe that if such bandwidth requirements are occupying near-to the full capacity of the link, and you start adding more of these bandwidth hungry robots to the network, problems occur due to dropping packets flooding the network with retransmits.

Maybe things could be improved by forcing camera information over a non-handshaked protocol like UDP, where no retransmission occurs if there are errors (after all, retransmitting an old camera image doesn't help, when there's a new image to transmit instead).

I agree that it would be really nice if HQ could get 6 of the 12, or all 12 Einstein robots, with the Einstein field, and test things out at HQ.

Even then, I'm not sure things would fail without the crowd of thousands of WiFi enabled devices inside a pseudo-faraday cage with it.

[Libby K]

You could tell that what happened on the field set the tone for the arena, the announcers weren't the same, Dean, Woodie, etc weren't the same and we were just seeing them via a webcast.

I'm not an engineer. I cannot speak to how the field works, or how it didn't during Einstein.

However, your statement struck me. I was sitting down in the front during Einstein, as I was speaking during the Awards ceremony.
Those watching on the webcast probably didn't see this:

Dean was down on the floor, behind the scoring table, during every match... trying to help figure out what was going wrong and how to fix it. He was ushered back to the stage for each speaker, but he wanted to help fix it too.

He was upset, the staff was upset... we all were.

As you have seen in the letter from President Jon Dudas... FIRST is going to try to figure out what happened and make a solution. I fully believe that it's important to recognize each of the Division champion teams for their efforts on Einstein in some way... and then work on resolving the connection issues for next year.

I had no bearing on what happened, but I am truly sorry it did. I'm sure FIRST will make sure this never happens again.

[JVN]

Whatever happened, has happened.

We as a community do not need to speculate on why it happened. We don't really even need to postulate on how to keep it from happening again. What we need to do is trust in the number of really great people who are working on it, and focus on moving forward in a positive way as a community.

How we as a community conduct ourselves now will define our collective character. Can we move forward in a constructive manner?

My personal thoughts are here:
http://blog.iamjvn.com/2012/04/reactions-celebrations.html

I had a great time at the FRC Championship, and I hope everyone else did too. Good experiences can happen even when things go badly. Let's make this into a good experience.

-John

[dag0620]

What we need to do is trust in the number of really great people who are working on it, and focus on moving forward in a positive way as a community.


I just want to say thank-you for saying that.

The Volunteers and Staff who deal with this aspect of the competition (FTA, FTAA, CSA's, etc.) have a very strong passions for what they do. These guys really want to get every robot working on that field, and it hurts them too whenever a team runs into problems, such as what happened on Einstein.

I personally have put my full trust behind them. I have never doubted there calls, and I feel others should do the same.

So I agree, as a community we need to let these people do their jobs, and give them our full support and trust, so they can Improve FRC and give us the best possible experience the can.

[AlecMataloni]

Good experiences can happen even when things go badly. Let's make this into a good experience.


I think a lot of people needed to be reminded of just that.

Oh, and welcome back, JVN!

[S.P.A.M.er 17]

My thoughts about Einstein. (http://www.chiefdelphi.com/forums/showpost.php?p=1164806&postcount=67)

[MooreteP]

I believe that this will be remembered as a watershed year for FIRST

There are things that will change after this year.
There are many intelligent people cogitating on the season.
We will engineer a better solution.

This was a great game this year. The winning alliance were teams with 15, 16, and 17 years in FIRST.
The universe conspired and after all is said and one, the result was inspiring.

[PayneTrain]

Everyone has a right to be upset that it had to come to a failure on the biggest stage in FIRST, with the founder and his second in command becoming as distressed as the teams on the field, for networking problems that have persisted since the inception of the control system.

That doesn't mean you should question the integrity of the organization we support so fervently. I don't question that FIRST is looking into this seriously, I just wish this was made more pressing before now.

[Karthik]

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division?

Yes. We found that it was very difficult to triple balance when we never had had three moving robots at the same time... In all seriousness, we desperately wanted to triple balance but the communication issues our alliance experienced made this impossible. Both 1114 and 2056 were dead for significant periods of all three matches, while 4334 was motionless for the entire first match.

---

The issues of Einstein 2012 were highly unfortunate. There's no sugar coating it. For the rest of my life, I will never forget the looks on the faces of team members of every team down there. Not just the losing teams, but the winners as well. No one was happy about what happened. We all felt so helpless. Everyone was trying to console each other, but there was nothing anyone could say other than "it's not right".

The most striking moment for me was when 2056's six year alliance captain, Isaac Hunter, looked at me and said "Karthik, can't we do something about this? We can't just let this happen." I was left speechless. I'm used to being the person who always has something to say to make people feel better. But at that moment all I could conjure was "There's nothing we can do. It's over."

The silver lining of this incident was the enduring spirit of gracious professionalism between the 12 affected teams. Everyone was looking out for each other, the lines between alliances became completely blurred. In many ways it felt like there was one giant 12 team alliance that won Champs together. I hope that gets recognized in some way or another.

The statement put out by FIRST after the event was very powerful. I commend Jon Dudas for taking such swift action. I'm confident that he will lead a thorough investigation into the events of Saturday afternoon. I hope that FIRST will prove my answer to Isaac to be wrong. I hope there is something that can be done to rectify what happened. In fact, hope is the wrong word. I know and have complete faith that FIRST is going to fix this. Let's just give them time.

In the mean time, I'd like to congratulate our 2012 Champions of the World, Teams 180, 25 & 16. These teams should carry that title with both honour and pride.

[Rlew488]

It was completely unacceptable

[Grim Tuesday]

I would just like to note that this thread started as "were the issues handled correctly" not "were the issues acceptable". I think everyone agrees that we would rather a working Einstein, and matches will all robots working. The question is, was there anything FIRST could have or would have done, given the situation?

[bduddy]

I believe that, on the day, FIRST (as far as we know) did what it could to fix the issues on Einstein. The real problem is, though, that these problems had been apparent for the past few days, weeks and even years, and FIRST and its top employees took every opportunity to blame communications issues on the teams until this issue made them finally realize that there was another problem. If they had actually addressed the issue earlier, this wouldn't have had to happen.

[EricH]

Given the overall situation (extended history of problems), no, the issues weren't handled properly.

Given the immediate situation on Einstein... Tough call, but I have to go with yes, under the circumstances.

Overall, the "it's not the field it's your robot" type of statements--and the fact that much of the time, doing something to the robot made the situation better--served to not get the problem addressed sooner. Overall, that's an issue. Sweeping a problem under the rug doesn't make it go away. It's better to face it head-on and try to run some tests--I know at an internship after some testing of a troublesome part we were able to tell a customer, "When properly set up, the whatsits perform as advertised. We see they haven't been set up properly in your situation." (We were also looking at possible alternative methods to perform the function of the whatsits in case of inability to set up properly.) If you think there's an issue, go for the root cause.

For the immediate situation, there are a lot of factors going into this one.
First, you have the published documentation. Replays are by field fault or true tie only under the rules. And you can't for sure say that this is a field fault (we're assuming that you're looking at only Einstein). So, by ruling it a field fault and replaying the first two semis matches, when there was no concrete evidence that it was a field fault, they did do their best to accommodate teams. Positive point.
Second, you have the big picture. Looking beyond just Einstein, the robots worked better on the divisional fields. This puts the weight towards it being a field fault, so more replays should have happened. Negative points.
Third, you have the schedule. The Championship traditionally runs over--the only question is, how far? By issuing more replays, you lengthen the time--and shorten the night out--and possibly force teams to choose between staying or heading for home in time to get some sleep before Monday's return to the daily grind. This balances out the big picture look, so you do limited replays. Positive point.
Fourth, the weather. In bad weather, staying inside is the place to be--and you don't want to be running to vehicles in a hailstorm. But, it has the potential to interfere with all types of electromagnetic radiation, including the field control system wireless. The thing is, most electronics will be built to handle that interference these days, because it's a known potential issue. So that's a moot point (other than the keeping people inside because of weather part--a positive but unrelated point).

With the scale being balanced like that, it's time to play "benefit of the doubt" and "20/20 hindsight" cards and say that FIRST handled the immediate Einstein issues as best as they could under less-than-ideal circumstances, but needs to be more proactive in looking into possible root causes when untraceable issues crop up in large-ish numbers.

[Peter Matteson]

Does anyone have a full field video that shows the scale of the issues?

From watching the web cast only I wasn't clear that everything I've read here was quite as bad as I've seen.

[JaneYoung]

The silver lining of this incident was the enduring spirit of gracious professionalism between the 12 affected teams. Everyone was looking out for each other, the lines between alliances became completely blurred. In many ways it felt like there was one giant 12 team alliance that won Champs together. I hope that gets recognized in some way or another.

The way in which those of you who were a part of the 12 team alliance are expressing how you dealt with and handled the pressure and frustration, is recognizing the qualities of champions in each of you. You are bringing this recognition to yourselves in ways that show true leadership and reflect the integrity of the experience. What happened was horrible. How the 12 teams responded was magnificent.

Jane

[dcherba]

It is not possible to change anything that has happened and under the circumstances everyone tried to make the correct decisions and behaved extremely well considering the drive and emotions present. We can't change what has happened but we can work together to understand the problem with facts and then design a solution. Those of us who have designed protocols and done in depth analysis of network traffic all know the performance is probabilistic and there are no guarantee performance metrics for this type of network. The cost advantage and ease of implementation for this hardware is significant and has to be balanced with the performance. There is no question that there are many hazards that exist in drive station software and also the CRIO basic design that were exposed by the level of network traffic that was present this year. There is a real difference between a design that works every time and one that works most of the time.
To solve this problem and move ahead we need to divide the problem into a couple of pieces.
First we need to find and remove the software hazards that make the driver station and the CRIO vulnerable to missing packet and possible lockup conditions. Some of these hazards may have been in the software for years but really were exposed by the level of network traffic this year.
Second we need to make a realistic model of the network traffic by in depth recording some trial events and looking at the options for network configuration that will improve the overall performance or issue clear limits that will bring the performance into an acceptable range. The quality of some of the wireless hardware may in fact play a major role in this analysis.
Let’s focus on finding solutions that after all it one of the life lessons FIRST is all about.

[Sunshine]

This.

100% correct. These problems were occurring on other fields, at other competitions, and other years with this control system. Something needs to be done about the system, not the implementation. The system they have is the problem, not specifically how they went about handling Einstein (though that could have been improved too, it's not the root of the problem).

Well said
Maybe we'll stop hearing how it is not the field and it must be our robot. The anxiety that Dean and Woody experienced was a small fraction of what the finalists and other teams have been experiencing.

[Astrokid248]

A quick disclaimer: I don't speak for the Robonauts, I speak for myself. Contents of this post are influenced by what I heard from my fellow mentors and students, but this is my opinion and you should not associate it with my team.

Well said
Maybe we'll stop hearing how it is not the field and it must be our robot. The anxiety that Dean and Woody experienced was a small fraction of what the finalists and other teams have been experiencing.

This is the real reason I am so upset. It's not just that we didn't get to play, it's that we didn't get to play and FIRST blamed it on us. "There was a problem in SF2-1; there may or may not have been a problem with SF1-1." That sentence, said by the emcee, seems like it was targeted directly at 118, at least to me. It sounds like they decided that we didn't deserve a second chance, but they were giving it to us only to appear fair. I was livid in the stands when I heard that, and I probably made my team look bad with some of my angry screams (for that, I apologize). But to claim it was our fault, after what happened in Connecticut and how hard we worked to replicate 364's issues at Lone Star, was just unbelievable.

Furthermore, the commentary during the matches made it sound as if we had control over how immobile our robot was. There's a reason we nicknamed Archimedes the Eh Team beyond them being Canadian; they were really incredible, and deserved to be thought of as the A team, the best FIRST could offer. But from the commentary of the match alone, you'd think Curie was playing against a bunch of rookies. It was very derogatory. There was nothing FIRST could do about the connection errors (at least, that late in the season) or the schedule or even the tornado. The show must go on. But where FIRST went wrong was making it seem like the teams were responsible, and then actively ignoring our plight during match commentary. It was rude, it was unprofessional, it was about as opposite as gracious as you can get. That's where FIRST abandoned its values, and that's what FIRST needs to correct.

We as a community will do everything we can to help FIRST get the communication issues resolved. We will overlook what happened on Einstein in the name of science, and continue to recruit and spread the values of FIRST. But I personally will not stop expressing my opinion in forums such as this until FIRST apologizes for the way they treated the 12 teams on Einstein. Kamen just knelt by the side of the field and looked upset while watching the train derail, and then pretended there wasn't a problem when it came time to pass out awards. You cannot tell me that was the right thing to do. FIRST has never needed accountability before, but they do now. 16, 25, and 180 deserved the win, but the rest of the alliances deserved to play.

[Solidstate89]

I'm really hoping that because these particular radio issues that affected Einstein were not only so debilitating but also so public (biggest matches of the season and all) that it will really, well for lack of a better word; "shame" FIRST into finaly looking seriously at the issue of connectivity. For as long back as I can remember (before they switched to routers) there have been connectivity issues. Years and years now this has been going on, affecting one team or the other with some field set-ups exacerbating the issue further.

It's just been a running joke, like the old submission process for Autodesk Inventor files (Pack & Go assembly - upload files - files fail to upload - contact Autodesk - Autodesk responds and says since the uploads failed for everyone they'd push the deadline back) until they changed the submission process for this season.

So, again. Here's hoping they take a very serious look at the issue. As far as some of the opinions expressed in this thread; I do think the fault lies somewhere with the consumer routers they're using and I do not think this is a "tainted" win. Every team had to deal with these connectivity issues. They never should have been an issue in the first place, but it happened and there's nothing we can change about that now.

[Undertones]

This could be a very long post, I will do the best to explain my full thoughts on the Einstein field issues.

Being the driver for 4334, I was front and center in experiencing this awful event. I can't tell you how awful it is for your whole alliance to go down during the championship finals. Forgive any bitterness I may have in this post, I am pretty biased on this event.

While I understand that FIRST had their reasons for handling things in the way that they did, it was completely disgusting. The volunteers that I dealt with were all less than helpful, they brushed me off more than once, and not once did they give us a straight answer. I realize and recognize that they didn't have answers, but the least they could do is admit there was a problem (which they did when we replayed those matches). It made it hard for anyone to believe that they fixed the problem when the problem continued. I was standing there listening as a volunteer tried to tell 2056 is was a battery issue with their bot. I mean, 2056 has had a perfect record this season in terms of up time and robot reliability. Loose connection in one match; improbable but plausible. Loose connection in 2 matches; highly unlikely but possible. Loose connection in all three matches; I mean, look at the stats and tell me that's what happened.

They did the right thing by replaying the first two matches - they did the wrong thing by powering through and denying that there was a problem. I don't think I'll ever forget the moment I lost faith in the FIRST system - just the injustice of what happened was enough to make me want to quit FRC. After a match when so many robots went down, holding faith that they would replay, and then not doing it... It's ridiculous. If there was a massive hole in the ice at the Stanley Cup finals, they'd stop and fix the ice. This was a world championship event, and the field didn't work? That should be the least of anyone's concern. And to think - if I won worlds, went home, got on Delphi, and saw that there was problems with the field, I would seriously question the legitimacy of the award. That alone should have been reason enough to fix the field. And think of how bad this made FIRST look - people who don't really know anything about the dynamics of Rebound Rumble and the structure of the FIRST ranking system/elimination processes must be thinking "Wow. How bad were the rest of the robots?"

As anyone involved in the community knows, FIRST is all about gracious professionalism, honesty, integrity... And they informed 10,000 kids by example that none of that matters when your reputation is on the line.

Now, I'm not going to go into "well, this would have happened if.." The All-Canadian Alliance has been the FIRST Canada dream for years now, and this year it became a reality. People recognized how strong an alliance we were. We had a really good run, and it stung to see and feel it end how it did. Anyone who's ever gone down during a match can agree with me it's probably the worst thing that can happen to you. It's horrible. And then for them to turn around and tell us it was our robots? Add insult to injury, why don't you.

Honestly, I'm impressed by FIRST's acknowledgement of the problem. I think we'll probably never know what caused it. I've heard all kinds of ideas on what happened, from thunderstorm interference to conspiracy theories involving the War of 1812. (I actually overheard someone say that last time there was a bunch of Canadians in the White House they burned it down).

So, as respectfully as I can, and with as much gracious professionalism as possible, FIRST screwed up big time. I hope they took away from this a valuable lesson.

I guess above all, I wanted to show the world our siiick triple balance. I think we can all agree it was a thing of beauty.

We need to make sure something like this never happens again. It's unprofessional, low, and unbecoming.

Learn from the past and apply it to the future.

On a lighter note, the rest of the championship was amazing. So good to meet so many amazing people, I had a great time aside from the last night.

Mac
ATA 4334

[sgreco]

In declaring a winner, you're also inherently declaring losers. How was it fair for FIRST to tell 118, 2194, 548, 1114, 2056, 4334, 233, 207 and 987 that they lost? None of these teams ever lost a legitimate round at the championships. This is the biggest question mark for me.

I realize I'm treading on thin ice for saying what I'm about to say and I want to preface it by saying that I have the utmost respect for 25, 180, and 16. They are very classy and professional teams, with incredible robots, and they all deserved to win a championship title. The problem is, the 118, 2194, 548, 1114, 2056, 4334, 233, 207 and 987 are all deserving as well. Nobody won or lost a legitimate match on Einstein, and I question FIRST's decision to award a title at all.

I really want to stress that in saying this I mean nothing against the winning teams. Given the decisions FIRST made, they followed exactly what they were supposed to do. They were even gracious and professional enough to hold back on celebrations out of respect for the other teams there. They were truly a class act. I want to also stress what I said in the beginning; FIRST inherently told the other 9 teams on Einstein that they had lost in declaring that one of the teams had won, and I don't believe this was fair to given the way the matches were played.

And I feel for the losing teams, but I feel even worse for the winning teams. They were put in a difficult situation, and they handled as well as anyone possibly could. They all worked so hard to win a title at the championships, and they deserving achieved their goal, but were unable to celebrate the accomplishment to the fullest. That must be an awful feeling, to know you won, and know you deserved to win, but have some form of an asterisk next to the title. I feel bad for them. They are great teams, and I wish them the best of luck next year!

[Kit Gerhart]

I think we all feel the same way, some more vocally than others. Our alliance felt reason to celebrate winning Curie, beating a very good alliance of Daisy, Poofs, and Air Strike. We did NOT feel so good about beating the Maple Leaf Alliance, because those were not fair matches.

At this point, we only need to trust that by next year, there will be a reliable control system, whatever it takes.

[jblay]

It seems to me like the teams on Einstein understand the spirit of FIRST more than the people that are running FIRST. All season these problems have persisted and teams were told it was their own fault and were robbed not only of the time they put into FRC but also the money they invested in the competition. The idea that it needed to happen on Einstein before something was done is truly disgusting to me. Today for the first time in my 7 years in FIRST I am not proud to consider myself a part of this great organization. Then again I am proud to associate myself with the teams that played on Einstein and acted the right way. FIRST should take a lesson from these teams, a lesson they were supposed to be teaching these teams not the other way around.

This is just my personal outrage because I have been waiting since week 1 to see how this game would play out on Einstein and now I never will.

[techhelpbb]

The volunteers that I dealt with were all less than helpful, they brushed me off more than once, and not once did they give us a straight answer. I realize and recognize that they didn't have answers, but the least they could do is admit there was a problem (which they did when we replayed those matches). It made it hard for anyone to believe that they fixed the problem when the problem continued. I was standing there listening as a volunteer tried to tell 2056 is was a battery issue with their bot. I mean, 2056 has had a perfect record this season in terms of up time and robot reliability. Loose connection in one match; improbable but plausible. Loose connection in 2 matches; highly unlikely but possible. Loose connection in all three matches; I mean, look at the stats and tell me that's what happened.

I've seen this over and over.

I've commented on it here (see post 204):
http://www.chiefdelphi.com/forums/showthread.php?t=104713&page=14

The problem is that if it is power quality issues they suspect, the field folk lack the tools themselves to confirm promptly and to some extent so do you.

So it's an easy fix all to move you forward. They tell you that it's some part of the power system. They have some authority so you naturally move forward.

Course when you realize that you've exhausted all your capability, you have not satisfied them, and you're still stuck. Then you start to get the impression they are messing with you.

I've not seen anyone recently make such a recommendation just to cause you problems to a team. However, if they did you'd never get past it at a competition....no matter how hard you try at the average competition I've seen you'll never prove your point with the tools on hand.

Sometimes I think that the field personnel when increasingly confronted over the issue naturally side with the familiar and assume that it's more likely it's your problem then the field. Their observation is just as anecdotal. You could have dropped a DC-DC converter after 3 matches as we did. Not a wiring problem in sight, but a bad D-Link AP power supply none-the-less.

In any case, neither the field folk or the teams should point fingers at each other. More often than not you didn't have the tools to remove the doubt, that doesn't make the concern invalid.

[thefro526]

In declaring a winner, you're also inherently declaring losers. How was it fair for FIRST to tell 118, 2194, 548, 1114, 2056, 4334, 233, 207 and 987 that they lost? None of these teams ever lost a legitimate round at the championships. This is the biggest question mark for me.



There was discussion while matches were being played on Einstein about giving everyone there a finalist medal. Most people there were instantly turned off to the idea since that meant there would be no winner.

Honestly, to me, that would've been more offensive than anything. I'd rather lose to the field and have a winning Alliance of 16, 25 and 180 than be called an Einstein Finalist along with 11 other teams.

[Undertones]

I've seen this over and over.

I've commented on it here (see post 204):
http://www.chiefdelphi.com/forums/showthread.php?t=104713&page=14

The problem is that if it is power quality issues they suspect, the field folk lack the tools themselves to confirm promptly and to some extent so do you.

So it's an easy fix all to move you forward. They tell you that it's some part of the power system. They have some authority so you naturally move forward.

Course when you realize that you've exhausted all your capability, you have not satisfied them, and you're still stuck. Then you start to get the impression they are messing with you.

I've not seen anyone recently make such a recommendation just to cause you problems to a team. However, if they did you'd never get past it at a competition....no matter how hard you try at the average competition I've seen you'll never prove your point with the tools on hand.

Sometimes I think that the field personnel when increasingly confronted over the issue naturally side with the familiar and assume that it's more likely it's your problem then the field. Their observation is just as anecdotal. You could have dropped a DC-DC converter after 3 matches as we did. Not a wiring problem in sight, but a bad D-Link AP power supply none-the-less.

In any case, neither the field folk or the teams should point fingers at each other. More often than not you didn't have the tools to remove the doubt, that doesn't make the concern invalid.
Thing is, we know it wasn't the robots. After every match we took them backstage and ran a full systems check, and they were all fine. And the bots weren't always down for the whole match. Just sometimes. They kept coming online and going offline without rhyme nor reason. We know it was not our robots.

[ratdude747]

There was discussion while matches were being played on Einstein about giving everyone there a finalist medal. Most people there were instantly turned off to the idea since that meant there would be no winner.

Honestly, to me, that would've been more offensive than anything. I'd rather lose to the field and have a winning Alliance of 16, 25 and 180 than be called an Einstein Finalist along with 11 other teams.

Perhaps everybody should have been given a "champion" medal, trophy, and banner.

[Undertones]

I've seen this over and over.

I've commented on it here (see post 204):
http://www.chiefdelphi.com/forums/showthread.php?t=104713&page=14

The problem is that if it is power quality issues they suspect, the field folk lack the tools themselves to confirm promptly and to some extent so do you.

So it's an easy fix all to move you forward. They tell you that it's some part of the power system. They have some authority so you naturally move forward.

Course when you realize that you've exhausted all your capability, you have not satisfied them, and you're still stuck. Then you start to get the impression they are messing with you.

I've not seen anyone recently make such a recommendation just to cause you problems to a team. However, if they did you'd never get past it at a competition....no matter how hard you try at the average competition I've seen you'll never prove your point with the tools on hand.

Sometimes I think that the field personnel when increasingly confronted over the issue naturally side with the familiar and assume that it's more likely it's your problem then the field. Their observation is just as anecdotal. You could have dropped a DC-DC converter after 3 matches as we did. Not a wiring problem in sight, but a bad D-Link AP power supply none-the-less.

In any case, neither the field folk or the teams should point fingers at each other. More often than not you didn't have the tools to remove the doubt, that doesn't make the concern invalid.
For argument's sake, let's say hypothetically that is what happened to 2056's bot. How do you explain the same thing happening to all the other robots that went down during the finals? I kind of don't think that all the robots had the exact same electrical problem.

[Undertones]

Well, seeing as there was no legitimate wins or losses on Einstein, it would stand to reason that everyone tied. However, I'm not sure if that would be more embarrassing for the teams or for FIRST. There is really no happy medium. Plus what would they do about visiting the White House and all that?

No matter how you cut it, it's the same garbage. The only thing we can do it figure out what went wrong and try to prevent it from ever happening again.

[ratdude747]

Plus what would they do about visiting the White House and all that?

AFAIK that is a CCW thing... so 1114 gets the honor this year.

[Astrokid248]

AFAIK that is a CCW thing... so 1114 gets the honor this year.
But they're Canadian, so shouldn't they visit Canada's prime minister or something?

[Gary Dillard]

But they're Canadian, so shouldn't they visit Canada's prime minister or something?

What color is his house? :rolleyes:

Googling last year, it looked like one student from each of the 3 winners went.

[Tristan Lall]

AFAIK that is a CCW thing... so 1114 gets the honor this year.
I believe they handle it on a year-to-year basis. Some years it hasn't happened, and the content of the invitee list has varied.

I'd rate it very unlikely that the White House would invite 1114. Rideau Hall, maybe. (David Johnston used to be the University of Waterloo's president, and so is probably acquainted with FRC, via the Waterloo Regional.)

[Undertones]

Makes sense.

And meeting Stephen Harper isn't really that impressive, quite honestly. He has a house in Calgary and goes to the movies and stuff with his family sometimes. Meeting him's really nothing to write home about.

It would stand to reason that his house would be red and white, however that is not the case.

I kind of think that this is a little off-topic now.

[Gary Dillard]

I kind of think that this is a little off-topic now.

At post #194 we'll cut you some slack; it seems to make it's way back on topic every now and then.

[cindycrews]

Robotics is the greatest organization in the world. I can't even imagine that FIRST will continue to leave the Einstein teams disillusioned, disappointed, &/or with the myriad of negative feelings they may currently possess. As Dean Kamen began to speak on Saturday, the crowds went wild! I said to my family, "Dean is the Rock star of Robotics!" I thought I was about to watch one of the greatest moments in my son's life, instead I witnessed his most disappointing moment to date. It's not about winning or losing, it's how we played the game, but it felt to me that Team 118, The Robonauts,(as well as the other Einstein competitors) didn't even get to play. I so wish we could recapture the initial excitement and redo the Einstein competition less the atrocities that played out. But short of Dean inventing a time machine, we cannot.
I challenge FIRST to some homework. Find the problem and a solution so no team in the future ever has to deal with the horror that these teams are enduring. This hasn't ended as many replay Saturday over and over in their head & continue to relive the event. They need adequate closure. Which leads me to another homework assignment: find a way to make it up to these teams. These teams have shown their gracious professionalism in an unbelievable manner and I have no doubt FIRST will do the same. An organization that stands for gracious professional won't possibly disappoint these phenomenal teams again....will they?

[Alexa Stott]

AFAIK that is a CCW thing... so 1114 gets the honor this year.

But they're Canadian, so shouldn't they visit Canada's prime minister or something?

I know this is totally off-topic, but that was one of the first things to pop into my head after "Finally!" and "YES!" when I heard 1114 won. :P:o

[demps45]

The communication issues that were seen on Einstein were not restricted to just that field. Our robot sat dead for two different matches on Newton and there was no explanation for the failure. We checked and double checked or connections and no proplems were found. We must assume it was a problem with the field. We were not the only team that experienced the same issues.

[kevincrispie]

Just as the championship this year was a learning experience for the students and teams, it also served as a potential learning experience for FIRST.

FIRST is not perfect. Don't get me wrong, I was very disappointed about what happened on Einstein this year, but the fact remains that FIRST is far from being the complete organization that many want it to be. In order to better the situation, FIRST, if they have not already, should embrace the notion of continuous improvement already exemplified by teams in its organization.

We all can learn form our mistakes, and FIRST is no different. Perhaps the reason why so many people are angered by the field issues is that we all have seen them play out all season, and on FIRST's biggest stage, Einstein, where the best and most inspiring teams in the world face off, the issues were significantly magnified. This, to me, is the signature issue. If FIRST wants STEM to be exciting and inspiring, they need to find a way to make more robots run, shoot, and score. There is little more uninspiring than toiling on a robot for 6 action filled weeks, working hard to earn a spot a championship, doing well, only to see your robot motionless on the field.

What's done is done. We cannot change the past. What we can do, however, is look to our future. The teams affected by communications issues should know that they are solid, if not amazing teams. Any team that had communication issues that may have affected their performance should come out fighting next year, and give the rest of the teams all they can handle in their quest for Einstein.

We all can learn. Even FIRST. Here's to the continuous improvement of the entire FIRST community--the students, mentors, sponsors, and yes, even the whole national organization itself.

-Kevin