Einstein Field issues Handled correctly?
 

Inferring nothing about the teams that played in the finals I believe the field problems were handled very poorly. I know there are time concerns but that was unacceptable. What do you think?


Edit: btw this happened to us (Code Orange 3476) twice on the Archimedes field.

Re: Einstein Field issues Handled correctly?
 

Agreed. They should have moved to another field and tried again.

Re: Einstein Field issues Handled correctly?
 

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

Re: Einstein Field issues Handled correctly?
 

That is also a problem... they need to keep at least one other field ready to go in case this kind of thing happens...

Re: Einstein Field issues Handled correctly?
 

They should have gone to Galileo after the first replay when 118 sat dead in the replayed SF1.1

Re: Einstein Field issues Handled correctly?
 

The other fields were not torn down, but even with that in mind, there really was nothing that they (logistically) could do. Add that to the huge storm, they handled it the only way they could. They couldn't move everyone, it would be a logistical mess. They couldn't change the field or change the red alliance station, that would take way too long. The only thing that I think they could have possibly done(and even this is a longshot, I think the FMS would prevent it from going smoothly) would be having the red and blue alliances switch so that they could make ONE HUNDRED PERCENT sure that it was the field, and not the robots*



*I'm not saying it was the robots, but this would make it so that there's absolutely no way they could say, 'It's you, not the field'

Re: Einstein Field issues Handled correctly?
 

I feel bad for 180, 25, and 16. Sure they were declared the winners, but it will forever be a tainted win. We will never know who the real winners should have been.

Re: Einstein Field issues Handled correctly?
 

I'm sure for next year, this is going to be on the list of "things to make happen."

For me, I can't be sure if they did the right thing or not. Under the circomstances, they probably did while in retrospect they didn't. Obviously there wasn't some backup plan in the event of the Einstein field having such errors, if it was the field at all. It might have been a "Einstein, the unsinkable field" mentality that it's been given time and time again, so having or needing a backup plan in case all goes haywire wasn't on anyone's mind.

That being said, as a volunteer myself I tend to give credit to the other volunteers and give them the benefit of the doubt that they did all they could. I won't say the field was to blame, but I would site the field as a probable culprit.

TL;DR, I don't know. But at least we know next year will have a lot more backup plans just in case of similar situations in the future, as well as a series of steps that would be taken to handle the issues correctly.

Re: Einstein Field issues Handled correctly?
 

Based on the situation they should have taken a break (30-45 minutes) to correctly review the issue and to allow the teams to test their connections. Just allow the teams spectating and competing to go break for a dinner break instead of just allowing the teams to suffer. How they handled the situation was very unprofessional as they did not even acknowledge after the replay any remaining issues and just remained quiet for an extended period of time before announcing the results.

Re: Einstein Field issues Handled correctly?
 

I don't know that I would call it a tainted win because we don't have any real information (other than our observations) to support that it is a field problem. Yes, this is the first time we've seen this kind of failure on Einstein but I don't think anything should be taken away from the winning alliance. We'll probably learn more soon...hopefully.

Did they handle it right? I'd say that they handled it well after the first 2 failures by replaying the matches. After that, I'm sure they did the best they could. No one wants to see robot bricks on any field at any level.

Re: Einstein Field issues Handled correctly?
 

Logistically, they did almost everything they could have. Replaying the first two semis was the only thing they could, I doubt from that point on they could realistically have replayed more matches, or used more time. They have a schedule to keep to.

However I do agree that they should have swapped sides, I've never been a volunteer but I imagine that's not too burdensome to change, and they should come up with a back-up plan in case of failure.

All in all it was the problem itself that was disappointing, not necessarily the solution. How they handle this problem hasn't been concluded, so I'd await further judgement. If they come out with some sort of explanation, it would help immensely.

Re: Einstein Field issues Handled correctly?
 

I may take some flack for this, so I'll try and explain my thought process as thoroughly as possible. I'm not yet entirely convinced that it's the field/FIRST's problem, and I don't think I know enough to make an educated call. I agree that it looks very suspicious, but this is how I see it.

These problems are very hard to diagnose the root cause of. There have been numerous long threads here listing as much evidence as we can, and many of the greatest minds in FRC haven't yet figured it out. These are not easy problems to solve, and I have full confidence that the FRC staff is working as hard as they can to work this out.

Many times, it is actually a robot problem. There have been loose connectors, USB issues, and more. After the CT Regional (which had its fair share of communication issues), I spent some time talking to the FTA. We agreed that some (but not all) of the problems were team caused - IIRC, there was a team that admitted its own fault on this. For the rest of the problems, the FTA said that he simply didn't know. There are so many variables constantly changing, and it's extremely hard to debug with the tools available. WiFi networks can be very tricky, especially regarding interference. I'm not saying it's FIRST's problem, and I'm not saying it's the teams' problem. Just trying to look at the big picture here.

Now, with a little more respect to today's Einstein matches, specifically. Yes, it is fishy that for some teams, this was the first sign of communication trouble. Yes, it is fishy that the robot at Red 2 was totally dead for 3+ matches in a row. Yes, this is a huge problem - just look at Dean and Woodie's facial expressions during their speeches, they both looked REALLY stressed out, and with good reason.

But if you were in their situation, what would you have done? Consider the weather: electrical storms and hail. That can't be good for the field networks. Now, I'm not in St. Louis now, but I think the inclement weather started right around the same time as the Einstein matches (EDIT: apparently, it started for the finals. But it's still possible for there to be electromagnetic interference before and after the hail started, since the storm was still in the area). Thunderstorms and field trouble are correlated, in this case (but this doesn't imply causation, however). But I think the weather played a large part in the trouble - the field network had to have had some problems with all the electrical interference in the air, from both the weather and people's devices in the stands. I would love to see some data from the FMS logs, DS logs from teams competing on Einstein, and any data about wifi traffic in the area. It may shed some light on the problems.

I do think they should have considered moving to another field or switching red/blue, and I hope they did consider that. But they decided not to, for whatever reason, and I remain confident that they chose this with a rational explanation; I'll give them the benefit of the doubt on that one.

Let's be frank: getting eliminated due to connection issues is just an absolutely awful way to go. There's no ifs, ands, or buts about it. It sucks. A lot. We see that we have a problem. Complaining won't solve anything. Let's look at the big picture, and try to see how all the many variables fit into place. The problem is complicated enough on its own. Let's put our heads together, collect as much hard data points as we can, and try to fix things.

tl;dr: I'm not going to pass judgement on how things were handled and whose fault it was. I wasn't in St. Louis, and I don't know all the facts that the field people knew. Until I know as much as possible, I'm going to refrain from saying it was either party's fault. Let's try to get as much data as possible into the open and work together to solve the problem instead of mudslinging on whose fault it was.

EDIT x2: I'm not saying that it's nobody's fault. It's possible that it is entirely FIRST's fault. But I don't feel comfortable making that call with the current information available to me.

Re: Einstein Field issues Handled correctly?
 

So let me get this right... The general consensus is that it was the Einstein field in particular causing the issue?

Because the way I see it, it was more likely caused by the large number of 3g phone signals creating massive wireless noise.

Re: Einstein Field issues Handled correctly?
 

180, 25, and 16 played and won on the red alliance. The field issues seemed to be just with the red alliance, so I would not consider it a "tainted" win.

I cheered when they announced replaying both matches. That was the correct decision, and as far as I know, the only decision they could have made.

Re: Einstein Field issues Handled correctly?
 

This.

Yes, the issues on Einstein were horrible, but after the first replay all you could really do is watch the train crash happen. The single most important thing is how FIRST addresses the problem in the coming weeks and months. If there isn't any comment or statement out of FIRST within a week or two, then I think we should all be highly concerned.

Re: Einstein Field issues Handled correctly?
 

Simply not true. Blue alliance had com issues as well. Although strangely they surfaced later in the tournament.

Re: Einstein Field issues Handled correctly?
 

That and the weather...

Re: Einstein Field issues Handled correctly?
 

I am curious as to how it would be possible for electrical interference in an enclosed dome where there was no apparent power surges or outages. Not to sound rude but is that even possible?

Re: Einstein Field issues Handled correctly?
 

This makes a lot of sense. To me, a field doesnt simply "go bad" for random driver stations at random times. It must have been an external cause

Re: Einstein Field issues Handled correctly?
 

I don't know if its possible or not, but after living in ST. Louis my whole life, I have seen this stuff happen during bad weather. Sometimes during bad weather I will lose wireless, without a power surge or outage. I wouldn't put it past the Midwest...

Re: Einstein Field issues Handled correctly?
 

Re: Einstein Field issues Handled correctly?
 

If only we could go back to IFI's radio system... wifi problems solved.

Re: Einstein Field issues Handled correctly?
 

This is how I see it. Feel free to correct me if I'm wrong. I'm not an expert on this stuff, but I don't think many of us here are. I'll do the best I can. Wifi networks use electromagnetic waves to transmit data (yes, oversimplified, I know). Thunderstorms emit lots of electromagnetic radiation. This interference, I think, had the possibility to cause some serious problems.

The dome is not impervious to electromagnetic radiation. Some interference can still get through (albeit less, compared to open space). Think about how, say, cell phones cause wireless interference: more electromagnetic radiation in the air, interfering with your network. This doesn't cause any power outages or surges, but still can take down the field network. It's similar with regard to that kind of atmospheric noise - it doesn't have to be enough for a power surge to impact the network.

Re: Einstein Field issues Handled correctly?
 

If this possible, maybe we'll see weather delays in FRC?! Like in sports, they could show highlights or replays of past games for the audience as they wait. :D

Re: Einstein Field issues Handled correctly?
 

What gets me is that

1. Einstein was never used. Do they ever bring out 6 robots to check a practice match or do they hope they assembled it right?

2. The robots were all in working order before coming to Einstein.

3. Some robots worked in some matches then died later others not at all.

FIRST couldn't find a problem, but there is a problem and when you are down to the final four you can't just decide to keep playing through matches when there is a problem like this.

What I would have suggested (please note I will admit I'm not an expert at the FMS or electrical components in general) is that they run a test match in between awards that bypasses the FMS system. Just like a team would if they wanted to run a practice match at home. Each head ref enables the robots in practice mode so they run autonomous followed by teleop. Allow the teams to drive around the field.

Everybody works fine and 118 moves then you have field error if someone dies and 118 is still not moving and robots loose comms then you have robot failure. If the field is at error I've always wondered why they could just run a match as listed above where the refs start each robot at the exact same time. Understandably you might have some robots running a tenth of a second longer than others but I see that as better than not at all.

Again the above is a real long shot but it is an option other than running matches and hoping everyone is okay because you can't find a problem. I was at the CT regional where 118 and other had problems and they traced them to USB and robot errors not the field and I didn't see that here unless they kept it quiet.

Overall very sad to see robots perform amazing in their divisions and not agree with that field.

Re: Einstein Field issues Handled correctly?
 

I would be curious to know where the Einstein field was during regionals. I'm treading very lightly here, because we had just come to terms with our season and accepted fate for what it is: a slippery eel. For those who just watched 118, you saw exactly what happened in the CT semis. Watching the Robonauts brick again, and again, was terribly sad. They have such a beautiful robot, and tripling with them was always picture perfect.

The field in CT was the same as the one in NYC , which also had persistent comm issues, predominantly on the red side. So, where has Einstein been?

Re: Einstein Field issues Handled correctly?
 

Einstein has been the backup field sitting in a warehouse all season, AKA never been used. Part of me believes that the robots we saw were the first robots ever to connect to that FMS. I don't know much about the system but I don't believe a FIRST field is working properly and is ready for competition until I see 6 robots play a match with no issues.

Re: Einstein Field issues Handled correctly?
 

There are a few problems here,wouldn't make sense that the entire stadium be electrically grounded in case of lightning striking the dome itself? Another concern would be based on the information provided why wouldn't wifi in schools or homes lose connection during bad storms? There is also the problem that the only way the noise would affect the robot connection would be if we all connected to that router since 3G and the competition signal run on different wavelengths.

Re: Einstein Field issues Handled correctly?
 

Einstein (correct me if I'm wrong) is the emergency backup field that spends all season sitting in a warehouse in Memphis, Tennessee.

Re: Einstein Field issues Handled correctly?
 

You are correct.

Re: Einstein Field issues Handled correctly?
 

The worst of the storm (hail and thunderstorms) came during the finals matches, which had the least communications problems. It's certainly possible, of course, but I don't see much of a link there.

Anyway, I think this question has to be asked: Why is the Einstein field one that has (generally) not been used at all during the season? Yes, I'm sure it's tested just as much as the other fields if not more, and I understand the appeal of playing the final matches on a pristine, unused field, but I think the benefits of using a tried and tested field outweigh those. If in a future year there are again a lot of communication problems, they could even make it part of the protocol to play the finals on the field that had the fewest such issues during its own matches.

Re: Einstein Field issues Handled correctly?
 

Yeah. Try and push a video signal over that!

Re: Einstein Field issues Handled correctly?
 

Wheeee. Well I feel a need to post a small rejoinder to plnyyanks' post above. My only quibble with his post is the suggestion that this isn't anyone's fault. This is unequivocally, without a doubt, FIRST's fault.

Understand, I'm not saying that the root cause lies with the field or something. I think it's pretty clear the the root cause of these failures must be associated with particular robots in some fashion. Galileo didn't really have any problems after switching to the Red station, after all. So it's unlikely that the issue is with a particular station color or number.

Nevertheless, it is FIRST's fault even if every single comms issue can be traced back to some specific problem with a robot. This is for the simple reason that we're playing with a FIRST mandated control system on a FIRST field, following FIRST's instructions for setting up and running on the field. All the equipment and software we use is supplied through FIRST. If they don't know this stuff well enough to troubleshoot these problems and help teams fix issues, it is entirely their fault for not doing due diligence and understanding their system. Yes, yes, a team or robot might be the cause of a particular case of comms issues, but it's inexcusable if FIRST can't tell that team how to fix the problem.

(shakes cane) Back in the day of IFI controllers, we didn't have these kinds of problems. When there were comms or controls issues, the IFI people worked the problem and solved it. The back in '07, IFI moved to new radios and teams had comms issues with them. IFI identified the problems and had FIRST post a list of fixes and solutions in a Team Update. Before the Week 1 regionals. Clearly, we have been in this situation before, only back then things got fixed because there were people available that knew the entire control system from code to chips.

The situation now is obviously different. I'm certain FIRST has experts available that know the NI hardware and software, PDB, etc. just as well as IFI knew things. I'm just as certain they don't have anyone that knows anything about our communication equipment since we're using off the shelf consumer components for that vital part of the robot. Heck, it took a FIRST team member/mentor writing a program to automate WPA programming back in '10. FIRST clearly has been treating our wifi equipment as a magic black box that should just work. It's entirely their fault for doing so, and we're seeing the results of it now.

Re: Einstein Field issues Handled correctly?
 

This: I'm not saying that an electrical current (like, electrons flowing) passes from the sky, through the dome, and into the field network. Like you said, I'd assume the dome is grounded, preventing this from happening. I am saying that the culprit might be electromagnetic radiation, not current. Electromagnetic radiation exhibits exhibits wave-like behavior as it travels through space. This would allow it to travel through the (grounded) dome and interfere with the network. The wave emitted by the storm would interact with the wave emitted by the router, get into conflict, and mess things up for the teams.

Re: Einstein Field issues Handled correctly?
 

Agreed... If that entire stadium isn't grounded properly I wouldn't want to be in it during an electrical storm. With the proper grounding that building makes a Faraday cage for everything inside it limiting (or eliminating) any electromagnetic waves from going in or out. This is the same reason that generally your cell phone loses signal quality inside large buildings like the stadium.

Satellite TV (well any satellite communications really) have issues with electrical storms because there is no way for that signal to avoid the interference the storm provides.

Regarding cell phones and other connected devices, generally they are running on different frequencies than the 5GHz (I'm assuming they were not using 2.4GHz) WiFi we use in FRC. While harmonics can be nasty and have an effect, generally they won't be strong enough to be an issue; they come more in to play in things like Space to ground transmissions (see LightSquared interfering with GPS despite being in a different frequency band).

There is no doubt in my mind that any issues were not caused by general electromagnetic interference with the WiFi system. I'm not discounting multiple WiFi networks being present confusing the routers on board our robots, but that is not directly related.

Re: Einstein Field issues Handled correctly?
 

Even if it wasn't interference, what happened to Code Bonde? Why couldn't they switch out to a Code Bonde setup and tried that?

Re: Einstein Field issues Handled correctly?
 

Here are my two cents for what it is worth.

We had around an hour after every field was finished before the Einstein field started. When the finals started on Einstein we had to listen to around 50 minutes of speeches and awards before we got to the first match. Was this really the best use of time in hind sight? Why can't each robot have a practice connection during that down time. NASCAR has practice time on the track, Soccer has warm ups on the field, Football has warm ups on the field before the super bowl. With the amount of time that this field sat in a warehouse how many hours of time were robots connected to the field. Every other field at the event had a previous regional where it was used and the equipment was tested. Second why can't one match be completed before the first awards and Deans speech are presented. If there was a comms problem there would have been around 50 minutes to troubleshoot the problem while Dean and others were speaking. Don't get me wrong I like listening to the speeches to a certain extent but I also think everyone is excited and want to at least see one or two matches right away before the speeches start. Had everyone seen there was a potential problem with this field before there is an issue where they are behind schedule due to Dean's speech or something else they could have realized they needed to cut the speeches shorter to rerun a match or something.

Just my two cents.

I think there will be a valid solution to this by next year that is stated in the manual. At the very least the the teams should have switched sides so that each team had to deal with the same field issues.

Re: Einstein Field issues Handled correctly?
 

Wow. I totally forgot about Code Bonde until now. Why DIDN'T they do that? I assume there must have been a reason...has it been used at all this season?

Re: Einstein Field issues Handled correctly?
 

I heard something about it being implementable at the regional level and not beyond that, but I'm not sure myself. Code Bonde was put in place for situations like this where wireless interference could be a problem.

Re: Einstein Field issues Handled correctly?
 

If wireless interference truly was the root cause, I'm sure they would have put it into effect. All of these events have a Spectrum Analyzer at them monitoring the levels at various frequencies. I'd imagine that everything on that front was well within tolerances.

Not to mention that it hasn't been used all season. Who's to say the problem wouldn't be made WORSE or you introduce other problems in the process of switching?

Re: Einstein Field issues Handled correctly?
 

If that's the case, then my point about the weather may be moot. This is news to me.

Re: Einstein Field issues Handled correctly?
 

Bonde was used once this season at FLR. Looking through the FLR thread, there wasn't much mentioned as to why. There also wasn't any comment on comms problems or anything, so go figure.

If it's implementable on a regional level, you should be able to run it on Einstein. They're clearly not running anything different on FMS at Einstein, judging by the fake Qual matches on the Match Results page. I suspect no one thought there would be a problem, and there wasn't time between matches to switch out the setup to the Bonde setup.

Re: Einstein Field issues Handled correctly?
 

I would think it could be due to the issues at FLR last year, which I believe are why Code Bonde was created.

Here's something else to note, in regards to the field being a fresh, never used field. Prior to the matches, the red alliance stations had the numbers, "16, 2194, 330". I don't know what the blue station had because of where I was sitting, but this struck me as something very strange.

Re: Einstein Field issues Handled correctly?
 

I think there was enough time between matches to switch this out. I mean how many awards were given out between each match. It may have been a logistics issue of not having the equipment where it needed to be in order to implement it, but there was enough time. My guess is that since it had not been used much the whole season that they left it in the truck when they unloaded the equipment.

Re: Einstein Field issues Handled correctly?
 

That's not true. We were on the blue alliance in the finals and were having major problems.

Re: Einstein Field issues Handled correctly?
 

Ooh, we have someone from one of the Einstein teams. What were the reactions of the teams with issues, and how did the FTA look at it?

Re: Einstein Field issues Handled correctly?
 

Look at the "Match Results" webpage. There's a fake Quals match with those numbers on the blue alliance (behind the red goals). I guarantee you they set things up on Einstein by setting up a new regional with the division winners and finalists, scoring some matches, and running a fake alliance pairing.

Re: Einstein Field issues Handled correctly?
 

I'm sorry, but I am from an FTC team...what is Code Bonde?

Re: Einstein Field issues Handled correctly?
 

Code Bonde is used as an emergency in case of a hostile wifi environment.

Re: Einstein Field issues Handled correctly?
 

Teams that lost because of it were livid. Teams that benefited from it shared the opinions of everyone here. I thought that the Curie eliminations were far more exciting than any single match played on Einstein. The finals on Curie were intense.

Re: Einstein Field issues Handled correctly?
 

But the problem is that based on the theory behind electromagnetic radiation we would have interference during ever major thunderstorm in lesser insulated buildings. Also if there was any major waves that would affect the stadium, other electronic devices such as phones and laptops would be affected as well.

Re: Einstein Field issues Handled correctly?
 

I didn't come on to rant just recognize something went wrong. If there were Com issues involved they should've took a better approach instead of in a sense saying oh, well better luck next year. I mean, what was the point replaying SF1-1 and SF2-1 then?

As far as the FTA goes, it seems like they didn't do anything to change the outcome or else it would've never happened. Or at least waited until the issue was resolved.

Re: Einstein Field issues Handled correctly?
 

Well, for what it's worth, as I was leaving the dome after Einstein, I was unable to send texts on my phone and got the error, "Network not responding". I have Verizon and not a smartphone, if this post even matters at all haha.

Re: Einstein Field issues Handled correctly?
 

You and everyone else trying to leave were using their phones, inside a pseudo Faraday cage, all trying to communicate with the same few cell towers. No surprise at all that the cell network could have been overloaded.

Re: Einstein Field issues Handled correctly?
 

I assumed it was something like that, as I said, that was a pointless post I made haha.

Re: Einstein Field issues Handled correctly?
 

in my opinion no one really won nationals. those disconnects cost the alliance the final matches. this could have changed the whole out come of the entire event. you should not pay upwards of 10,000 plus dollars and be left in the dust. they should have taken a 1 hour time out and finished the awards. and then finished the matches. leaving those teams behind is not the spirit of gracious professionalism at all.
that's just my too cents on the issue

Re: Einstein Field issues Handled correctly?
 

I completely agree, splatter. The amount of issues with the FMS was unprecedented, especially given that many of the robots have had nearly no FMS issues until Einstein.

With the problems going on, they might not have been able to do much of anything in an hour. I don't know how long it would've taken to enact Code Bonde, but it should've been tried.

Re: Einstein Field issues Handled correctly?
 

our team had connections problems all day luckily we were able to work them out but the FTA did every things in his power to help us he did not leave us flat once. and that was in practice matches and early qualifiers. so it was very disappointing i really felt like no one truly won at all in a general sense. but nice job to all of the teams in the finals nice job all ! to find a true winner it should all be re done but it wont happen. good job winners!

this is a reply to

I didn't come on to rant just recognize something went wrong. If there were Com issues involved they should've took a better approach instead of in a sense saying oh, well better luck next year. I mean, what was the point replaying SF1-1 and SF2-1 then?

As far as the FTA goes, it seems like they didn't do anything to change the outcome or else it would've never happened. Or at least waited until the issue was resolved.

from jm987

Re: Einstein Field issues Handled correctly?
 

Like I said, "seemed to be". The only bots that were dead on the field were on red.
From the stands, it's difficult to tell when a team is having comm issues, unless they stay put through the entire match. :(

I think FIRST needs to take a good hard look at their system, and find a way to implement something more robust. Perhaps the field should remain pristine, but use a different (used) FMS.

Re: Einstein Field issues Handled correctly?
 

from what i remember the robot status light (the orange blinking light)
was in the non connection state of the non moving robots.

Re: Einstein Field issues Handled correctly?
 

233 (Curie, blue) was dead in F1.

Re: Einstein Field issues Handled correctly?
 

Missed that. Thanks for the correction.

Re: Einstein Field issues Handled correctly?
 

The first part is incorrect. A flashing light at the corresponding alliance station indicates loss of communications. It was in this way that I and other mentors on my team observed comms issues. For me, at least, it was extraordinarily clear that the entire Curie alliance suffered from comms issues at some point during finals. The only two robots to maintain comms in finals, I believe, were Bomb Squad and Raider.

A different FMS is an option, albeit a logistically impractical one. The better solution, imo, to extend on Kevin's post, would be for FIRST to at least have demonstrated throughout the season (via team updates or some manner) that they were actively working to isolate and resolve the issue. It also should have been clear that pre-match testing was absolutely imperative given the prevalence of these issues throughout the season.

FIRST's unwillingness to explain these issues was also unforgivable.

Re: Einstein Field issues Handled correctly?
 

This, I agree with.

This, I do not. If the FMS is at the root of the issue, why is it remaining in place? Would you solve the issue of termites creating lots of sawdust in the kitchen by buying a better vacuum?

If the TCP/IP protocol is to blame, it's time to move on to a more reliable technology.

Re: Einstein Field issues Handled correctly?
 

I think most if not all problems that occurred could have been prevented if the connections on Einstein were tested earlier in the day. If the problem had indeed been the field, they could've spent the rest of the time up until the finals troubleshooting. If there were no problems before but problems later, the bots should've been check then to see if it was the bot's fault. If not, they definitely should have looked into radio interference and storm interference which they had the equipment to do so (at least for the radio part anyway). I don't think it was so much as an inability to find the issue as the lack of will and patience to do so.

Re: Einstein Field issues Handled correctly?
 

TCP/IP as designed and properly implemented is a reliable protocol. If a packet is not received properly it will be resent. Is it right for FIRST? Is the implementation throughout the system correct? I don't know.

Re: Einstein Field issues Handled correctly?
 

Something to keep in mind:

118 has had communication errors a couple times this year. Yes, there were clearly problems on Einstein that effected a lot of teams. However, once they worked on the problems and replayed the matches, it was only 118 that never moved. I don't think we can blame the entire thing on FIRST, especially since we don't have all the details.

Re: Einstein Field issues Handled correctly?
 

In regards to the weather possibly playing a factor into interference with the WiFi:

I'm by no means a radio-frequency or weather expert so I just had a conversation with some guys in the meteorology department at my school and they were skeptical about the storm being a factor. Lightning strikes do produce radio waves called sferics which would make their way into the building, however they are usually in AM frequency range, and shouldn't cause interference with the devices that FIRST uses on the field.

They participate in storm chasing during the spring and use plenty of wireless networks during the process, so they have plenty of experience with wireless devices in the middle of large thunderstorms. While I'm sure there are some equipment differences, they haven't ever had interference problems in the middle of a storm, nor heard of anyone else having those issues. Of course this isn't a 100% comparable situation, but it's relatively close IMO.

Re: Einstein Field issues Handled correctly?
 

It might help to switch to the 5 gz bandwidth for wifi. Some laptops would need to have an adapter for the network. Very few devices use the bandwidth and it would cut down on interference.

Re: Einstein Field issues Handled correctly?
 

For what is worth. One thing I was told by the FTA at an off season event. The signal to start Autonomous or Teleop is only sent once at the beginning of that state. I personally have no idea how the field communicates with the robot but this statement makes sense for why some robots don't move. If you only send a signal once and in that split second while that signal is active there is a glitch in the WiFi (which happens) then the robot misses the signal and won't ever activate that particular state. However this is not how TCP/IP is meant to work. If a signal is missed there will be a message to send the signal again for a few more attempts to help create the full packet. It seems to me that this is not happening. When a robot is not communicating with the field there are separate signals being sent to flash lights above the driver station. Why can't a simple protocol be created that while this light is flashing there are multiple attempts to tell the robot what state to be whether it be Auto or Teleop. Having a one shot attempt to tell the robot to go seems like it is leading to these problems.

How much do we know about how the field works? Can this hypothesis be proved invalid?

It also seems that when the FTA is looking at packet transfer there are packets that tell the robot to be disabled till the match starts and then there are packets after the match starts that have the data from the DS but how many packets tell the robot to switch from disable to Auto and then to Teleop. When there are only a few packets missed this does not seem like a lot to the person watching the graphs and what not on the FTA table but its not about how many packets are lost its about which packets are lost. Does the FTA have a way to tell that the packet to switch from disable to Auto and Teleop has been received properly? Can the robot tell the field that it has received the packet properly so a status flag can be used to light and led on there comms station? I agree that telling a team they have no idea what is happening is not acceptable. Anytime you say you have no idea what is happening it just means you are either A) not taking enough data points or B) you are not taking the data points on the right data.

Re: Einstein Field issues Handled correctly?
 

@Ruskin: we DO use 5GHz for the robots.

I propose a move to the licensed 3GHz 802.11 band.

Re: Einstein Field issues Handled correctly?
 

Does anyone know which frequency bands the robots use? Could weather radar be messing them up?

Re: Einstein Field issues Handled correctly?
 

Is it possible that since Einstein was the back-up field that if there was an Update on the FMS for every-field, Einstein Never got that update, thus causing all these Robots to have a problem?

Proposal for IRI

All alliances come to IRI and we play the Real Einstein matches to try and find who the winner should be?

Re: Einstein Field issues Handled correctly?
 

well there's an idea nobody I saw thought of before.

Re: Einstein Field issues Handled correctly?
 

5GHz 802.11a WiFi

Re: Einstein Field issues Handled correctly?
 

While this problem has been brought to light by the Einstein matches I do not think it is new. At MSC 33 died in the semis about 3 seconds into teleop. In the finals 67 had similar looking drop outs although I cannot confirm that it was in fact an identical problem. 1114 also seemed to have issues although those could be on the robot side (Both 67 and 1114 were connection issues.) Perhaps we, as a community, should begin documenting all of these failures throughout the season in order to identify patterns and the actual quantity of failures. It is easier to accept there is a problem when you have data to back it up and with many examples of failure to look through it would hopefully make identifing the problem that much easier.

Regards, Bryan

Re: Einstein Field issues Handled correctly?
 

Weather radar uses microwave radio pulses that would not have any effect on other wireless signals. Otherwise your internet would drop every time the radar makes a rotation (radars run at all times, not just when there is a storm)

Re: Einstein Field issues Handled correctly?
 

There are several channels / frequency bands.
E.g. 36, 40, 44 (5180, 5200, 5220 mhz). If they use the middle channels (unlikely, but you never know), the radios are required to listen for radar and switch channels (called Dynamic Frequency Selection).

Re: Einstein Field issues Handled correctly?
 

Does the cell phone noise hypothesis hold any water? I feel like these things happen when there are a lot of people near a field, such as during elimination matches (see 1717 on Newton for similar problems). A lot of phones including mine have WiFi capabilities and may have interfered with the routers on the field.

Re: Einstein Field issues Handled correctly?
 

I was at the event where 118 was having issues (CT) and it was traced back to their robot.

Whatever this was really cost the championship for every alliance. They all were strong and each could have taken it if they were running on full power. 987's alliance was extremely close in the last match the High Rollers only needed to score a few more but they were motionless.

Re: Einstein Field issues Handled correctly?
 

For complete thoroughness, specs on NOAA's NEXRAD radars say they use 2.7Ghz to 3.0Ghz. NOAA also says they're constantly emitting high energy pulses. Short high energy pulses. Something like 1000 pulses per second, but the transmitter is still only active about 7 seconds out of every hour. So it seems unlikely that that's the culprit.

Re: Einstein Field issues Handled correctly?
 

Am I correct in thinking that the radios use spread-spectrum techniques? It has been quite a few years since I have worked with such radios.

Re: Einstein Field issues Handled correctly?
 

Okay, somethings wrong here. In the last 4+ hours since championships have been over, there have been 14 posts in the 'congrats 16, 26 and 180' thread. There have been hundreds complaining about the field issues. Whether you like it or not, they are the FIRST World Champions. It isnt their faults that there was a problem with the field. Their achievements are just as important as field issues. I suggest that everyone heads over to the thread to congratulate them.

http://www.chiefdelphi.com/forums/sh...hreadid=106040

Re: Einstein Field issues Handled correctly?
 

Holy crap. I didn't know about that.... I think that's part of the source of the problems.

I'm pretty sure FMS sticks each robot on a different 5GHz channel, and actively manages them, in a way that *may* not be compatible with DFS.

If the consumer grade d-link detects 5GHz radar stuff in its channel, and decides to try to jump onto another channel and FMS doesn't play nice on that decision guess what? boom, robot dead.

Would also explain why Red 2 in particular was having issues. Guessing the closest radar antenna was on the frequency that Red 2 was arbitrarily assigned by FMS.

Re: Einstein Field issues Handled correctly?
 

I have witnessed competitions as a FRC competitor and as a scorekeeper at a regional.
I have spent some time thinking about what they could do to fix/prevent the problems occurring. First, they should have used an electronics system from a field that has been through regionals and has proven to work in a competition.
Then, they should have tested robot connections before matches, possibly going as far as to run a match with the robots sitting on their carts, not on the field, and the sounds turned off so the audience doesn't hear.

If they didn't get solved with the earlier ideas, they should have started changing the field electronics out incrementally starting with the portions that interface with the robot, and prepping a second server to try as a complete overhaul and start the electronics from scratch. Then they should have tested connections during awards and speeches.

Replaying the first two matches was the right thing to do, but when problems persisted, they should have tried to do more, if for no other reason - because this is EINSTEIN.

Re: Einstein Field issues Handled correctly?
 

This was what I was thinking of:
http://en.wikipedia.org/wiki/U-NII
Where radar avoidance is mentioned several times.

Re: Einstein Field issues Handled correctly?
 

This is what sums up my feelings. You could tell that what happened on the field set the tone for the arena, the announcers weren't the same, Dean, Woodie, etc weren't the same and we were just seeing them via a webcast. The fact that nothing was done to remedy the problem is flabbergasting when you are in the most important matches of the season.

118 might be a seperate issue, but you don't have robots just start to lose comms in matches all at once. I could see 1114's issues come back but have robots that performed 100% in their division elims hit Einstein and all but three experience issues you have a MAJOR problem on your hands.

Re: Einstein Field issues Handled correctly?
 

Pretty sure our radios operate in the 5.15-5.25GHz range. From D-Link's website:
As a result, per U-NII our radios are not subject to DFS.

Re: Einstein Field issues Handled correctly?
 

In the future, it may be advantageous to operate in that band, specifically because most consumer grade wifi does not.

Re: Einstein Field issues Handled correctly?
 

Blaming TCP or IP should be completely out of the question. First of all, these are two of the most (really, THE most) used networking protocols in existence today. They are used by every single device (that's billions of devices) that connects to the internet.

Furthermore, one of TCP's advantages over other transport-layer protocols (such as UDP, commonly used for tasks such as video streaming) is it's reliability.

Which brings me to...
Though I don't know for sure, I'm assuming that the FTA was telling you that the application only sends the signal once for auto/teleop, constantly for disabled. Even though the application layer only sends one message down to the lower layers (TCP, IP, etc), TCP will keep sending the packet over and over again until it receives an acknowledgement. If that packet is dropped, it will be resent.

I very highly doubt that. I also doubt that never having used the Einstein field before would be to blame for any issues. The hardware and software are identical to that used on every other field.


Are there kinks that need to be worked out? Yes.
Do I think it is the FMS or field hardware's fault. Not really.
Do I think it is the robots' fault? Not really, though I have seen enough robots to believe that robots with various electrical or programming issues are just confuddling this problem more.

Re: Einstein Field issues Handled correctly?
 

This.

Re: Einstein Field issues Handled correctly?
 

Well the only problem with this theory is that DFS is an Access Point feature. The Access Point picks a channel to transmit on based off its detection of possible interference. I don't know exactly how the FMS manages the wireless access points, but I think the wireless channels are manually selected. Even if they're not, the DFS wouldn't kick in that often, and the WiFi clients shouldn't have trouble following the AP from one channel to the next. But mostly, I think FIRST is setting channels manually so this wouldn't be a problem.

Re: Einstein Field issues Handled correctly?
 

I think only the field and game elements are pristine, and the electronics have been used at other events.

Re: Einstein Field issues Handled correctly?
 

I don't know for certain, but Einstein is a complete field and as far as I know they use that field in its entirety for those matches.

Re: Einstein Field issues Handled correctly?
 

Don't have anything to add except to say that this is not true. The problems in CT were never diagnosed. We swapped out a cRio in Houston because we saw the same thing in a practice match, and couldn't bear the thought of doing nothing, even though the cRio swap didn't make much sense. We had no problems after that swap until what you saw on Einstein.

Re: Einstein Field issues Handled correctly?
 

Bill has always implied that the emergency field in Kentucky (?) or Tennessee (?) was a total emergency field in case something ever happened to another one; otherwise it was just for Einstein. However, I would consider an electronics system that was in use for a few events to be more solid than an untested one... at least we know where and when it has and hasn't worked before.

I imagine/blindly put faith into the idea of it will be an awkward topic of discussion in a blog post from the director.

Also, while I will congratulate the teams that have one (and they do deserve it) I believe it's just as important, if not more, we solve the issues relative to the complex and mentally/emotionally taxing 4-year-old communication system.

Re: Einstein Field issues Handled correctly?
 

I agree. These communications problems are not new. They've persisted through out all levels of competition to this point.

As I've stated elsewhere when Team 11 had issues 3 times in Philly. I didn't feel that it dramatically effected our placement based on what we saw when we competed entirely free of issues at other events. By a turn of events we went to Championships anyway.

Sure there's room to argue the point. However, as long as the people are effected fairly randomly let's not forget that these folks built some really phenominal robots and we all played the same game.

Chance is always an element. Be it a bizarre unexpected balance, a sudden broken part or even a software failure. Sometimes the chance is a spectacular show and sometimes not.

As far as how FIRST handled this? Troubleshooting of this type has been limited to the events. Everything else, like this survey, is generally a collection of anecdotal evidence. I expressed my concerns about field communications before this year's first event (based on our off-season experience at Monty Madness last year...see the topic about alternate control systems) and I expressed my concerns about how the troubleshooting was pushed back generally at all events in the 2 recent years with this radio system in the other topic. I myself was spare parts at the MAR Mount Olive events and we had a few issues with this problem and despite much effort to track the only thing that helped them was driving with the camera off and hence mostly at a disadvantage. The situation was put on this path by FIRST and people close to FIRST. This was the risk at every event and every event had the same problem not just the Championship. I presume that FIRST had figured they had this problem well under control when they started Championship given the pattern of the behavior. Perhaps FIRST just felt it fair not to eliminate the problem now that it had already messed with all the events up to that point...but they did allow some replays which is inconsistent (perhaps the story changed under the hot lights). Things apparently did not work out. I saw the look on everyone's face. They don't need this topic to tell them this can not be allowed to happen again....we need quantifiable evidence not anecotes and we need it when the problems happen not after the fact when things are all apart or it's just speculative no matter who's speculating really. We as teams can't test on their competition fields; both because we don't have access to them all the time and because often the rules prohibit the effort during competition matches.

I'm sure it'll get fixed it's hard to ignore it's a priority under the circumstances.

Re: Einstein Field issues Handled correctly?
 

What I would do with the Einstein field is to run it in a Week 0 event before putting it into storage. It won't be as pristine, but you get a chance to clean it up before the big show, and you know that it worked (or didn't) when you put it in the trailer. I'd rather have it working and slightly ugly than pretty and not working.

Re: Einstein Field issues Handled correctly?
 

Another idea relating to this. Take the electronics from the most reliable field. AKA has the least comm failures throughout all the regionals. But take the carpet game pieces and field elements from the backup field. Then you have the best comms with the best looking field. The backup electronics could be used at MAR or another state district to get them ready for CMPs. Maybe not the best solution for the backup electronics but it better than them being tested at CMPs on Einstein.

Re: Einstein Field issues Handled correctly?
 

We would all also love Einstein with maybe a few short speeches.

Also, I will state one more thing before I turn off for tonight.
I came into FRC as a freshman. You know what my first memory of a regional was? Not starting play until the afternoon in 09 because the field was bugging out. You know the thing a remember most vividly in 2010 outside of our ludicrous finalist bid and the case of the missing in-game piece? Standing in queue for 45 minutes in VA while we dealt with field issues. 2011? Watching a robot get pulled off because of communication issues the field was experiencing (the station where the team was located had problems all afternoon with a variety of robots). 2012? Almost losing in the semifinals because the robots would lose communication midmatch and we would reboot midgame to stand a chance.

I know chance is a factor in everything, but when I pick up a remote, I don't plan on blowing up my TV. The chance of that happening is severely mitigated by evolved technology.