Last two matches of Logomotion on Einstein
 

Was the same system used on Einstein last year (Logomotion)? If I remember correctly, during the last two matches of Logomotion a RED Alliance robot worked in autonomous and then did not work at all during driver control. Was this a robot problem?

Re: Einstein Field issues Handled correctly?
 

The radios use a spread spectrum technique for transmitting the data at the most fundamental level. Various techniques are used to enhance the data throughput from what is achieved in the 802.11a,b,g to get the 802.11n data rates.

One of the benefits of using spread-spectrum techniques is the very high noise immunity giving a highly reliable connection. More reliable than narrow band transmission techniques. Instead of transmitting a signal over a narrow bandwidth, the information is spread over a very wide bandwidth, at a relatively low amplitude. The receiver de-spreads the received signal, restoring the original narrow bandwidth and high amplitude. Narrow-band interference sources (radar, noisy electrical equipment) would be squashed in amplitude and spread over a wide bandwidth by the de-spreading process in the receiver and the majority of the noise energy can be filtered out very simply. That is one of the reasons why this technique was used by military radios for many years before the ISM band was made available for civilian use.

Due to how the spread-spectrum transmission technique works, it is difficult to believe that only one or two channels can be corrupted for a whole match by a narrow bandwith, short duration transmission like the weather radars or wideband weather phenomena such as lightning. They ought to affect all channels in a similar way, simultaneously or not at all. The only way I can think of to adversely affect a single channel would be to jam it using a spread-spectrum transmitter set to the appropriate channel. Either FIRST missed some signal source when setting up the fields or someone was intentionally jamming. If find the latter hard to believe.

The radios are a consumer grade item but their design would have been tested very thoroughly by the manufacturer and their other customers. I would imagine that the teams experiencing trouble would have been able to check for connection before each match or they would have raised a concern and not allowed the match to start.

This leads me to ask if the FMS software that manages the robots through the 802.11n link may have some bugs in it. I am sure that the developers of this software would have tested it to the best of their abilities. However, there can be issues and conditions that only show up with certain combinations of hardware and other software. The FMS software would have far fewer users and instances of use, overall, than even low-volume commercial software so obscure bugs can be very difficult to trouble shoot and would probably require tools not available to the FTA at a competition. Perhaps the developers should visit one of the upcoming off-season events and bring all their tools and toys. That is what our engineers at work do when a customer experiences unusual behaviour that the Field Service people cannot fix.

At Alamo, one of the robots in the alliance opposing ours was immobile for the whole match leading to the match being replayed. If my memory serves me correctly, it was announced that 148 had become indicated as a no-show so their channel was shut down, rendering them immobile for the duration of the match. I have no way of knowing if our (148's) experience is at all related to what was seen at the CMP but the symptoms look similar. By the way, one of the other teams did all the scoring and handily beat our alliance single-handed :(

Re: Einstein Field issues Handled correctly?
 

Did the thunderstorms supposedly start around the same time as the beginning of Einstein?

Re: Einstein Field issues Handled correctly?
 

One other thing that I'd like to say.

Einstein this year was a minor speedbump compared to another event a couple years back. Admittedly, said event had a number of factors besides the field issues complicating things.

I would not say that we've had nothing but problems since switching the control systems/wireless. However, I will say that field connection problems seem to have become more prevalent, and if this continues to be an issue, I expect that FRC HQ will be under significant participant pressure to change something with the system. Whether it's the FMS (which was mostly reliable the last couple of years), or a "Not my system" attitude held by both teams and field crew on occasion, or the routers on the robots, something needs to be made more reliable.

With the IFI system, they could tell right away whether it was the field or your robot, and if it was your robot they'd help you find and fix the problem. I haven't seen that type of full-system support the last few years--maybe because it's a multiple-manufacturer system. The system as it is is pretty good... but with this public of an issue, it could use some improvement.

Re: Einstein Field issues Handled correctly?
 

I think this is a very plausible theory. TCP is a "reliable" protocol, which means it retransmits afet some time period if the packet is not acknowledged. However, it only does this for unicast (point A to point B comms) packets. But multicast (point A to many points) or broadcast (point A to everyone) do not have acknowledgements and don't retransmit.

Does FIRST use multicast or broadcast in this application? I don't know, but it makes some sense that they would. By using one of these, you can be sure that nobody gets started before the others, assuming everyone gets the packet in the first place.

Re: Einstein Field issues Handled correctly?
 

I'm glad to hear you state this. This is what I had heard through the channels, but I had yet to hear it firsthand. I recall the announcement at CT, that there was "no way that the field could have caused a robot's cRIO to reboot." I subsequently talked to about a dozen engineers who said (while never saying "that's not true") words to the effect "there's no way you can make such a blanket statement as that."

I'm split when it comes to answering the OP question. There is not much else the field administrators could have done. To put it as simply as possible, there IS an issue. The issue has affected teams that have the BEST software and engineering minds. The problem/s remain elusive and diagnoses have been numerous and varied.

Administration needed to make a decision, and they did not have all day to do so. It looked like a no-win decision.

Robonauts, we all ached for you. We hope we get to see you triple-balance at least one more time!

Let's not forget to congratulate the champions.

Re: Einstein Field issues Handled correctly?
 

I would blame the preparation more than anything. There's no excuse not to have the field ready for FRC use. I'm more curious than anything to see what FIRST has to say: what the problem was; if they diagnosed it; why they didn't move to another field; why they replayed the first few matches, but then one after that; if the weather impacted their decisions; etc.

I'll give them the benefit of the doubt until I hear an explanation, but I really hope it's a good one. I think FIRST owes a good explanation to all the teams, volunteers and spectators.

Re: Last two matches of Logomotion on Einstein
 

We linked this problem back to the case shorting BB-775 that we were using for our arm. However last year during qualifications on Archimedes there was a blue driver station that was giving teams communications problems throughout the day ... us included.

Re: Einstein Field issues Handled correctly?
 

I believe we need to go back to the videos, the full and unclipped videos, if such things exist, and look and see exactly who had issues and when they had issues. I was not there and the webcasts were focusing only on the robots that were functioning, so attempting to see who was not functioning was... near impossible.

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division? I have been hearing of definite issues with something on the 118 scene, and slight hints at something with 1114.

Absolutely true and beautifully said. The first step is who, what, and where. The details, not the speculation.

Re: Einstein Field issues Handled correctly?
 

Honestly, I would rather compromise time for a better, more fair matches.

Re: Einstein Field issues Handled correctly?
 

Here's a minor suggestion that may help test Einstein in the future:

Teams that are eliminated early in QF's from their respective divisions could go do a 'test run' match with some stipulations (stay off the bridges so they stay shiny, the balls may be horribly scuffed, etc). That way Einstein field can be tested with 32 different setups, including those with camera traffic, custom UDP sockets, and "nothing but dashboard" screens.

Re: Einstein Field issues Handled correctly?
 

It appeared that during SF 2-1 or 2-2, each robot on the 2056 lead alliance had some time when they were unable to operate, whilst each robot was able to move at other points in the same match.

Re: Einstein Field issues Handled correctly?
 

They could have run a match swapping red & blue. That may have helped debug, and identify if it was a field or robot issue. They could have also asked that the teams involved ship their robots back to HQ instead of home, so the engineers could try to recreate the failure and set up the Einstein hardware with these robots again.

Now that the field is broken down, and the robots that were involved are gone, all that they can do is look at logs and wonder if they have the pertinent data.

Re: Einstein Field issues Handled correctly?
 

Not trying to be condescending here or anything, but I didn't see any representative of any team so far post what they personally tried to do to remedy it.

During one of my team's matches, we were motionless, but after we restarted the cRIO during the match, we got comm's and code and were perfectly fine.

Did any of these Einstein teams (esp. 118) restart/reboot their cRIO during the match?

-Duke

Re: Einstein Field issues Handled correctly?
 

I think they did the best they could in the time they had.

I would've liked to have seen them move to matches on Einstein sooner. That could've given them more time to troubleshoot the problem by putting more of the speeches and awards in between matches.

I'm amazed that this seemed to be a mystery. I work on telecommunications equipment. Part of my job is analyzing logs of communications failures. Those logs will usually pinpoint problems pretty well. Perhaps that's just because the FCC has set the bar so high. Compared to a worldwide telecommunications network the setup for the field shouldn't be that complicated. It seems to me that FIRST probably needs to add more troubleshooting and monitoring capabilities. The good news is that they control the software in the field, they control enough of the software in the robot, and they control enough of the software in the driver's station to do quite a bit. I think they probably need to add some more capabilities inside the robot, like have it log the battery voltage for the entire match inside the cRio. Ideally the system would be able to detect common robot failures that are completely unrelated to the field. That could really help improve the quality of the "it's not the field" message given to teams. I think we all want to see the best quality matches. I wish I could help more.

Andy

Re: Einstein Field issues Handled correctly?
 

Good thought, but the Robonauts practice shop is adjacent to a major airport.

Further, they are also within a few miles of the NWS office in League City Texas, which has a doppler radar. That's got to be closer than the radar that covers the Edward Jones Dome.

Therefore, I think it is unlikely that they would not experience problems in their practice building, but would at the dome.

Does anybody really think the Robonauts would allow such problems to go unresolved if they could duplicate them in their shop?

Re: Einstein Field issues Handled correctly?
 

How exactly would we go about rebooting our crio mid match when we have no communication with our robots? Walk out on the field and cycle the main breaker?

We will share our full thoughts and experiences in the coming days.

Re: Einstein Field issues Handled correctly?
 

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.

Re: Einstein Field issues Handled correctly?
 

I may be a bit late on this (I only read up to like page 4 so far, I've been on a 15 hour bus ride back.) but I did check the wifi signals in the dome while Einstein was going on. I only picked up about 4 signals, only like one or two were somewhat strong.

Also one question that arose on the ride back from a few people: Why would the teams agree to play on a broken field?

Re: Einstein Field issues Handled correctly?
 

I was sort of thinking about that earlier...

Back when I was in high school, I was giving a demo at a basketball game. The score board was programmed wirelessly from a table on the side of the court.

When I drove the robot in front of the table, the robot died... I had to push it out of the area of that table, and it regained control.

This was using the IFI system, on the 900MHz radio, but it seems like everyone is moving their stuff to 2.4GHz nowadays.

Good thought!

Re: Einstein Field issues Handled correctly?
 

FIRST did what they have done for four years when these kind of issues have occured. Give maximum one round of replay and then proced with the matches regardless of the result.
These actions are not fair to the teams on the field, winners and losers, it is unfair to the teams that competed and have already been eliminated, but most importantly it is not fair to the kids who worked so hard all season just to have it come to an end like that.
As great it is to say that we use this cool crio from NI, it is not worth this. I fear that spreading the word of FIRST has become the word of FIRST instead of teaching kids life lessons. What happened on Saturday was a result of a decision made by FIRST 4 years ago that was not in the spirit of FIRST.
It is my opinion, and I know I am not alone, that it is time for a serious change to the control system because enough is enough. It is also my opinion that FIRST needs to remedy what happened on einstein by awarding the championship to all 12 teams on einstein, but this is something that I don't think there is a chance they will do.
A big move needs to be made by FIRST here and if it isn't done I suspect much fallout in the frc community with more teams just going over to do vex.

Re: Einstein Field issues Handled correctly?
 

Once again, I'd like to kindly remind people the problem is most likely not the NI C-RIO. The C-RIO's job is to process the information given to it from the sensors and inputs from the radio and use that to control the various aspects of the robot.

The problem most likely is the radio. The C-RIO does not connect to the driver station at all. It's the D-link radio that does.

The C-RIO was introduced 4 years ago in 2009 to help in handling the vision processing that was introduced more heavily in 2009 than any other game, and to accommodate for the old IFI technology. The radio, on the other hand, wasn't chosen too well, and as a result left many teams without connectivity to the field.

tl;dr: It's most likely the radio's fault, not the C-RIO's.


EDIT: I am making these accusations because while it certainly is possible it is something else, multiple robots in every match had comms issues. Some of the teams never had comms issues before. I don't have a field diagnosis, but this is my result from troubleshooting the problem with what I have.

Re: Einstein Field issues Handled correctly?
 

As a mentor I think that the current control system (the entire control system including the radio) offers the opportunity to teach about troubleshooting. The problem is that while the opportunity to learn troubleshooting appears the tools, access and freedom to do that troubleshooting does not.

When I worked as the spare parts person at MAR Mount Olive I brought my oscilloscope. It surprised me the gulf between the teams that actually used it and the teams that neither knew what it was or even had any idea of what to do with it on a FIRST robot. The difference in levels of understanding was not between the veteran teams and the new teams were you'd think it would be.

I think that FIRST needs to very carefully reconsider their process for troubleshooting. Otherwise I suspect that the only thing a highly integrated control system does is remove students from the equation in favor of placing the responsibility directly on FIRST.

I respect either aspect of the matter. I'd just hate to make FRC operate on the equivalent of the LEGO NXT system (and I'm not knocking LEGO NXT I have been an FLL judge for years).

Re: Einstein Field issues Handled correctly?
 

Hello all,

Now that the dust has settled, if only a tiny bit, I'll share what I saw from Einstein. You may know me as Will, I work with team 207's strategy and drive team. I am a FIRST Alumni and have been mentoring for 4 years now. let me make it clear that Team 207 is home and nothing but thankful to team 233 and 987 for an amazing time at world champs and our first trip to Einstien. Through the chaos that was the Finals, they showed that they are pure class and of the highest examples FIRST has. (341, 254,and 78 as well)

Yes, many, an unforgivable number in fact, of robots were either dead or experienced significant periods of losing coms. Some, clearly game changing. This sucked the emotion and prestige of the finals, clear out of every team on the field. Not one of the drive teams, or pit crews in our alliance cheered as we made it into the finals after semi's. It was a feeling of disbelief, that FIRST was allowing such a thing. It would have taken 30 minutes, to replay all of Einstein back to back in my mind. but that did not happen.

As a programming mentor for Pink said earlier, this is not how ANY FIRST team wanted to win or lose. I remember being on the drive team myself and dreaming of one day coming back to mentor my team there, to feel that rush, to cheer to the absolute maximum the human lungs can. To have been dead silent, and not even be able to rally our team between F1 and F2 because everyone's sentiment was, whoever works longer will win, is the largest disappointment I have ever had in my FIRST life. Even if making it there after an EPIC curie division Finals. From what I see in finals videos and scores, 341's alliance in the division finals was as competitive if not more so than any alliance on Einstein.

However, the most important feeling and experience to get away from this, is the spirit of FIRST teams. They are completely separate from fields and external chaos such as sever weather, curses, coms issues and all else. Every team on Einstein was united in their dislike of what happened, and treated each other graciously and professionally. When 987 needed time to inspect their robot and wanted to call a timeout (and I still don't know why we were denied one) 180 came to offer any assistance and tried to call theirs as well. both finalist alliances congratulated each other, on the field, and in the pits. The students were proud and even 118, and the Canadian teams did not hesitate in spreading hugs and handshakes to us all. The students themselves began making plans to replay all matches at IRI with robots in the same condition as they left the field Saturday. Dean Kamen was on the field, as was Dave and the look on their face said it all. They were disgusted and Dean was almost in tears as a Girl from a team I cannot remember pleaded the cause of her team. Dave was gone it seemed by the time the finals were over, and Woodie's face from the stage looked like he was viewing somebody drown and could not help. I insist, the truest qualities of FIRST were never seen in such bright colors and they were here. Technology failed us and perhaps the field crew have a huge catastrophe to correct. These finals will be remembered as the day the field chose the winner. BUT IN NO WAY, SHOULD THESE EVENTS TARNISH THE IMAGE OF FIRST, ITS MISSION, ITS STUDENTS, TEAMS, OR FOUNDERS.

Some of the greatest Robotics matches I have ever seen took place on Curie field in the finals. And I speak for every member of team 207 when I say we are beyond proud of all we did, and all our alliance partners did. We would put money on our alliance anyday, and if we lost a fair match, we'd cheer so loud for the alliance that beat us that our lungs would explode. Regardless of what trophies or banners the fields gave us, the embraces, laughs, comments, and connections with amazing FIRST-ers are sooooooo much more valuable. I truly hope the Einstein replay at IRI happens, not to play these matches, as much as to share them with such amazing teams and individuals, like we should have been able to.

I hope a lot changes next year, but only as far as the field goes with technology, and in organization and error management. the Human aspect of FIRST and it's competitors is something we can be prouder today of than ever.

-Will

Re: Einstein Field issues Handled correctly?
 

Be careful making blanket assumptions and statements like that without any evidence supporting it. This is really a general statement... we can all speculate as much as we want but I am not aware of anyone (possibly even FIRST) that has hard data that points to one component or another. Anecdotal evidence seems to be pointing many in one way, but there are multitudes of layers to our complex control system in which any one of them could be the cause.

Re: Einstein Field issues Handled correctly?
 

If it is so simple then why didn't FIRST make a change in radios after the immediate issues of week 1?

Re: Einstein Field issues Handled correctly?
 

It's not that simple.

Some robots have power issues.
Some robots have radio issues.
Some robots have code in the cRIO that causes communications problems.
Some robots have bad components.
Some robots have bad wiring (that sometimes doesn't show up unless it's under load or getting impacted just the right way).
Some robots have issues with video swamping.

I can say that I personally have used the NI cRIO 8 module PLC in industrial use and it's a nice product. A little expensive but nice.

This said there are many, many examples of the D-Link AP's power connector being picky and a possible source of issues. That could be fixed by soldering wires for power to the PCB inside the D-Link (if only FIRST would allow it or provide some like that at each event).

It's not fair to say that we have sufficient quantifiable evidence to lay blame on any one piece of hardware. Good troubleshooting dictates that quantifiable evidence must exist for this to be more than speculation.

Re: Einstein Field issues Handled correctly?
 

I honestly hesitate to answer. Maybe switching fields just simply wasnt an answer. It is hard to say for sure, but I anxiously await what FIRST might have to say about this. I certainly would hope that this gets solved for next year. Still, congrats to The Bomb Squad, Raider Robotics and SPAM.


In a responce ot this, what were they supposed to do? I see little that the teams could do.

Re: Einstein Field issues Handled correctly?
 

Thanks for sharing your thoughts. It's better to post a link instead of re-posting the exact same text to multiple threads (especially for lengthy messages).

Re: Einstein Field issues Handled correctly?
 

"This is where gracious professionalism comes into play." Quote from my son Daniel, one of the drive team members of 2194 Fondy Fire. I have no doubt that the FIRST organization will handle this well. I saw Dean Kamen squatting by the field, looking at things, with concern written all over his face. And I am very gratified to see that the posts here on Chief Delphi are so supportive. You are all awesome and we are still happy beyond words to have been the first Wisconsin team to make it to Einstein. :D Our most sincere thanks to our Alliance members 548 Robostang and 118 Robonauts.

Re: Einstein Field issues Handled correctly?
 

I own a theater company and I manage a few theaters out of my local hometown and use a very similar setup to what they use in the rigging on Einstein. Wireless DMX is very behind in the amount of channels it can support and the lights that move, the intelligent lighting takes up many channels as every single aspect takes one channel. It would not have made any sense for them to have used wireless DMX. Using W-DMX would also mean flying all of the dimmer packs for the lighting, then the rigging isn't balanced on one side if those packs are up there.

I have used 2.4GHz and 5Ghz for intensive applications such as media streaming in theaters that were using wireless DMX at the same time and experienced no interference.

Re: Einstein Field issues Handled correctly?
 

Agreed. Using wireless DMX would be way too much work for this kind of setup as well as I thought I saw some DMX wires on the metal bridges they were using to hold everything up. Getting all the lights to work correctly is enough work without factoring in wireless well at least for me.

Re: Einstein Field issues Handled correctly?
 

Well, I'm in no way an electronics guy, so I may not have worded this correctly. However, there's something you can do on/through your driver station to do something similar to "restarting the cRio/radio/something". We had no comm's or code and we clicked something on our Driver Station during the match to fix it (I was not the one doing this so I'm not 100% sure what they did). If someone knows what I'm talking about and could explain what I mean further, that would be awesome.

-Duke

Re: Einstein Field issues Handled correctly?
 

There is a button to soft restart a connected robot's cRio. What you're describing likely didn't affect the cRio.

Re: Einstein Field issues Handled correctly?
 

I have been a student and FIRST mentor since I was 8 years old. I am now 21, and with school and work now consuming my life, I will be announcing my resignation from the team that I have grown to love and hold as my number one priority over the last few years. I have grown with this team from the "new-beginning" where we lost all sponsorship my freshman year, and paid mostly out of pocket to build and compete in the FIRST Robotics Competition.

Since this was what I considered the "last year" I decided to dedicate most of my time, mentoring and helping build both an amazing team, and robot. Along with the a dedicated team, thousands of hours, and a hard earned budget, this was the first experience I have had on Einstein. I can remember watching all of the great teams that I grew up with on Einstein from 1998-2011, and dreaming that with enough dedication, fund-raising, and overall team effort, we would one day be able to compete with those teams. Our team worked to find sponsors, mentors, and build a dream robot. After many all-nighters and extreme dedication, finally, that opportunity came with a near perfect alliance of 118, 2194, and 548.

Though it is still unknown what the root cause of these issues were, I am amazed that with a overall financial contribution as large as those in which large corporations and other donors have contributed, along with the number of engineers and other volunteers involved, FIRST was still unable to fix an issue that was known week one.

My "pit-crew", along with others, sat field-side disheartened by the fact that technology was failing not only our alliance partners, but also all of our opponents. An image that sits burned into my brain is the "This is FIRST" video that used to be on the FIRST website, where one of the mentors says "Months of Work.... Sat there.... Dead...." and really brings an emotional side out of me that can't be put into words. I can say with a large level of confidence, that this issue had nothing to do with the robots, but rather the field. I think the scores of these Einstein matches were a clear indicator of how messed up the outcome of this really was.

A team like 118, who is mentored by engineers that work on projects that literally leave our planet, who had a robot that not once had an issue through our entire Newton competition, without even a slight modification in code, were handicapped with a robot that "Sat there.... Dead...." every match on Einstein with the only reason being that of field coms.

118 proceeded to be lectured by an FTA that it definitely was not field issues, but rather issues with their robot. They pulled up a report from their robot only to see that the field could definitely be the reason for the loss of communication at the start of the match. The FTA said it could be their cRio having an issue, so they swapped out to a new cRio only to have results matching those from matches prior.

I was amazed how teams on the field pulled together, worked as a team, and almost every team said "we did not want to win this way." All teams were upset at one point or another with the outcome of these Einstein matches. From the field, we heard the lack of cheering, and could see the look of utter shock in the faces of everyone in the audience. Though the FTA's did work hard to help solve this issue, I believe that FIRST had no idea what was going on. Do I think they made the right decisions in how they handled the situation? Far from it. Do I think they knew what the issue was? Absolutely not. But after that entire disheartening fiasco, I would assume they wont let it happen again. I think that all the teams were amazing, and though it was upsetting, it was an honor to leave FIRST surrounded by everyone I grew up admiring.

Congrats 16, 26, 180.

Re: Team 2194 Fondy Fire
 

Congratulations from one of the Newton finalists!

I am in a minority that think the issues were handled correctly, and in accordance with current FIRST policies. After the first two matches, they did exactly the right thing (and somewhat outside precedent) with replaying them. The next matches, there were scattered problems, but in accordance with what is de-facto policy, they did not replay or cancel them. They did not move to Galileo because it was already torn down. They did not crown twelve teams Champions, because to do that would be to admit defeat completely and utterly. FIRST never does that. It appeared that the issues were fixed when 4334 began working.

Sidenote: We were one of the backup bots for the Newton Division, so our drivers were back behind Einstein if anyone has an questions about what was actually going on. We all know the saying about assumptions.

In the first finals match, it didn't appear from my position in the pit next to Einstein by the VIP section that there were any field issues. It is clear that there were in the second match, but we have to think about the enormous pressure here. Was the decision deferred to the head ref? Did Dean Kamen make the call himself? Everyone is talking about how rustled they looked, and I can attest, Woody looked somewhat put out when he was reading the awards. Dean looked nervous and taut while standing next to the field. There is a human element here, and the stress and pressure of deciding a world champion is enormous.

Regardless, if the FTAs could detect no problem with the field, then they made the correct call in not replaying the match. It follows all precedent. I covered the reasoning in not declaring all teams World Champions above. That isn't to say that the field issues on Einstein were disgraceful.

The issue was not with how the problems were handled. They were with the field/control system. It means that there are quite a few things that need to be greatly looked at, and fixed before next year:

• There have been many high profile cases of very high tier teams being unable to connect to the field (118 and 1676 among others). This needs to be fixed by next year, especially the fact that some fields were able to be connected to and others not.
• Einstein needs to be tested. It is supposed to be the best field in all the land, and if there are problems on it, it becomes a massive debacle (ie, this thread). Someone suggested running a practice match on it with divisional quarterfinalists first. This would diminish the 'virginity' of the field, as well as reduce the value of "being on einstein". I suggest that the GDC brings the 6 bots they build to test the game when they're making it and run the tests that way, being careful not to scuff anything. They can do it quietly whenever they please, maybe Wednesday night.
• The idea of Einstein needs to be reexamined. It seems somewhat stupid to put all your hopes on an untested field. Why not play the finals match on the field of whichever division won last year?

I think we need to thank all the Einstein teams for remaining Gracious and Professional, role models for the whole community.

Re: Team 2194 Fondy Fire
 

My friend and I examined solutions on the 6 hour ride home:

-Putting the finals on an existing field poses the logistical problem of seats. The teams on Archimedes (if that were the chose field) would have the advantage over every other division when it comes to getting good seats for "Einstein". Also, how would FIRST block off seats for the Einstein teams? You can't block those out for all of Saturday if they're in the middle of a division's block of seats, and you can't really kick teams out of those seats in time for "Einstein".

-Moving fields during awards (after the issue was noted) is a logistical nightmare. The podium, cannons, judges seats, student rep seats, etc are all set up to be where they are, and you can't just move that in a quick and organized fashion. Not to mention stampedes of people trying to move seats.

Anything else we've said has been mentioned and scrutinized already.

Re: Team 2194 Fondy Fire
 

I see your point, but nobody wanted to see what was going on anyway with robots not moving, and basically illegitimate eliminations on Einstein. They could have finished the awards while the teams move, then transfer to Galileo once they didn't need the podium anymore. At this point, who cares if every spectator was able to get a good seat? It's better to have legitimate elimination rounds then to have everybody disappointed watching illegitimate ones.

Also, I'm not sure how the screens worked, but it was probably reasonably easy to get a feed from another field displayed on the screen at Einstein. That way, not everyone has to move.

Re: Einstein Field issues Handled correctly?
 

I wasn’t sure which thread to post this in, but I figured this was the most appropriate. For those who don’t know, I was a mentor with SPAM from their rookie season until 2006 and I was on stage in Einstein with them this weekend. Three of the current mentors on the team were students on the team when I was there, they have returned after college as professionals, and are shining examples of the best that FIRST has to offer. I am not going to say that I speak directly for them or any of our alliance, but I’d like to share some of my thoughts and emotions.
First, I think our sentiment is best summed up by the way Karthik and I greeted each other on stage at the end of the event. I said “We’re having kind of a tempered celebration” and he replied “I understand”. We would have replayed matches all night if necessary; when we were told we were replaying match 1, no one even batted an eye. We didn’t want to win this way, but it is what it is. We consider ourselves World Champions; we felt like we had an unbeatable alliance, but I’m sure the other division winners felt the same way. Although our celebration was somewhat muted, our loss of elation pales in comparison to what I’m sure the other division winners are feeling about what happened. After we lost the championship finals in the third match in 2002, I replayed that match over and over again in my head –if we had only done this or that we might have won. What are the Robonauts or the Pink team going to imagine that they might have done differently? My heart goes out to you.
As far as the OP’s question, I’d like to mention part of what the head ref Dr. Aidan Brown told the alliance captains before the final four started (as relayed to me by our team captain, so I’m not quoting and I don’t remember the exact words). He said this is a competition, but it is also a show. (He also told them to have fun). I fully understand this, and the show must go on, so some decisions for the benefit of FIRST and the ceremony had to be made at the expense of the competition. I think it was a lose – lose situation, and this was the best they could do. This could have been prevented long before Einstein, but that is a discussion for other threads which we have all been following all season. I saw one FTA in tears afterwards, and I know everyone involved was doing all they could. 10 years ago when we were on Einstein, it wasn’t part of the closing ceremonies; we just played in front of whoever was still there in the afternoon, and we could have replayed without much fanfare, and then the closing ceremonies were in the evening. When they switched to the current format, I thought it was fabulous and wished we could have had that experience in ’02. I still feel that way, so I guess this is the compromise we have to accept. Thankfully, a lot of the corporate leaders were at the division eliminations matches up close and got to experience the thrill of the competition, and hopefully their support will continue so that we can advance the mission of FIRST.
There were some discussions amongst the teams on stage about ways to make a public demonstration, including the winners declining to accept the championship and having all 12 teams go up as finalists. This discussion took place after the semi-finals and before the finals, so we didn’t know yet that we would be the winners (in fact, we knew we would be the red alliance and we were worried). I went back and forth on it, weighing the need for a statement versus the ramifications to FIRST. Our alliance wasn’t unanimous in what to do, so we decided to just accept the results and proceed as normal.
As far as having the teams go to IRI and play it off (and I haven’t talked to anyone on SPAM about this, but it’s just my 2 cents), if you’d like to write the $10,000 check to SPAM for them to travel, and cover the mentor’s lost wages for another 2-3 days missed at work, maybe they’ll be able to get the paperwork through the school system in the month they have remaining to allow an out of state / after school is out trip to take place. I’ve only been able to make to IRI once, when my daughter wanted to visit Purdue as a possible college and we made the trip coincide with IRI. The rest of the time I have a real job and a family that I like to spend time with and work to do around the house to catch up on from the 2-3 months I neglect it during build and competition season. Not that the team wouldn’t love to attend IRI (for all I know they already had it in this year’s plan and budget), but it doesn’t just happen on the spur of the moment.
Finally, thank you for the respectful and courteous way that this discussion has taken place; pretty much everyone has caveated their comments with “not taking away anything from 16,25,180”, etc., and that means a lot. FIRST is a big family and we‘ll get through this together.

Re: Team 2194 Fondy Fire
 

I don't want to get in a back and forth, so this will be my last post (and we can PM if you'd like to continue conversation).

Finishing awards while teams move: drive teams? teams in the stands? That's pretty disrespectful to the teams winning awards and the speakers presenting them. No one feels too great about themselves when they're talking/being presented and the audience is too preoccupied with something else.

And teams not moving but the matches moving? You can't stop people from moving and I'm pretty certain 80% of the FIRSTers I know would want to move to actually see the robots.

I'm not saying I'm glad things turned out the way they did. I couldn't quite get excited about Einstein after the issues weren't fixed. I'm just saying that FIRST didn't have too many options and had legitimate reasons for not switching fields.

However, I wouldn't mind them presenting all the awards while field crews and FTAs switched all the FMS components of the fields (have one crew gut Einstein's FMS and another gut the most reliable field and then swap). I would gladly take a delay of any time span to fix the problem (and I'm sure you're thinking "well then we can just switch fields", but that's a hairier mess, in my opinion, than switching field elements).

Re: Team 2194 Fondy Fire
 

This'll be my last post about it too. I meant just move the drive teams, robots, refs, field crew etc and keep the crowd where they are until awards are given out. Then when they're done with the podium and the presentation, let people move. It would present some logistical issues with everyone moving at once, but I feel it would have been better than having illegitimate matches played on Einstein. People have enough class that if you told them to wait before they move, they'll wait. Then you put it on the jumbo-tron in case not everyone got a seat at the new location.

Re: Einstein Field issues Handled correctly?
 

We were on the blue side and died for an extended period of time in one match on Einstein, and briefly in others. At least one of our partners died for a time, and had glitches. We had no such problems on the Curie field. Regarding who "should have won," we'll never know for sure, but if played on a properly working field, it is quite possible that the Canadians would have beaten us, and we never would have made it to the finals.

As far as solutions, this year is done, but for next year....

1) One of the division fields could be used as Einstein for the "superfinals." It would have been well-tested after running over 150 matches. To my knowledge, we had no field-related issues on the Curie field during 3 days of play. If i'm wrong on that, please correct me.

2) Teams could play most of their matches on their "home" field, but be rotated through the Einstein field as was done at Disney, and maybe since then. That way, the field would be well tested, and presumably fixed, if there were problems.

Re: Einstein Field issues Handled correctly?
 

"Dear FRC Teams:

Thank you for your incredible enthusiasm and Gracious Professionalism throughout the year and at the Championship.

We apologize for the technical problems that affected the final matches at our Championship. We will examine all of the facts, report our findings and ultimately solve any and all identified issues.

Sincerely,
Jon Dudas
President, FIRST "

Just received this, posting it here in case anyone doesn't receive it.

Re: Einstein Field issues Handled correctly?
 

I agree.

Work through the OSI model on both ends of the link.

http://en.wikipedia.org/wiki/OSI_model

It's complicated...but by process of elimination I'm sure the problems can be found now and in future developments.

Re: Einstein Field issues Handled correctly?
 

As much as everyone would like Einstein to be in "pristine" condition, I think this is a good solution moving forward. It doesn't change the format of play at the end, and it has a very good chance of working out field issues (and if the FMS needs to be updated/replaced, we'll have ample warning).

Re: Einstein Field issues Handled correctly?
 

For those complaining that Einstein's prettiness would be lost in test matching before finals, They could use used electronics and virgin game components. They are separable, Our team uses the game components without all the electronics (like score counters and displays from the walls of the station). It's not like you can tell whether electronics have been used before.

Re: Einstein Field issues Handled correctly?
 

I believe I may have a relatively Simple Solution to prevent such an issue from happening again* - well actually two ideas on the same train of thought.

What if we were to Give each Alliance an Introduction Match on Einstein? When an Alliance is sent to Einstein, they make it a point to introduce them either via Video or by having them walk onto the field. Why not give each alliance the field for 2 minutes and 15 seconds and tell them 'Show 'em what you've got.' This serves a variety of purposes:

-Each Alliance Has what's essentially an uninterrupted Practice Match on Einstein. They can check Camera Calibration, Code, Tweaks, Communications, etc. If each robot runs during this intro match, you've now ruled out a bunch of machine issues from causing a robot to sit dead.

-These intro matches are going to be AWESOME for spectators. Imagine 2056, 1114 and 4334 on the field, uninterrupted driving the score up. It would be an EPIC sight. This gives everyone watching a good chance to get a feel for who's going to be playing on Einstein.

Or, Why not play Division Finals On Einstein? Basically, we'd be turning Einstein into a Full Elims Bracket with the Division Finals now serving as the Einstein Quarters.

Kit, I believe you are correct. As far as I am aware, the only team that had multiple issues with their robot not running on the field was 971, and the issue was on their end IIRC.

78's Dying in Final 1 was due to the radio coming unplugged.

*The real solution to all of this is to make a 100% reliable field, but that's been covered enough here.

Re: Einstein Field issues Handled correctly?
 

I'm a bit worried that many people are writing this off as an "oh its cause they didn't test Einstein" issue. Really? Did you hear about 1717's issues in their division?

The problem here is deeper. These issues persist on many FRC fields and this needs to change.
Ultimately the firmware and hardware we are given by FIRST needs to be built tougher and respond more reliably.

Everyone contributes far too much money and time to be met with faulty reliability on FIRSTs end.

Re: Einstein Field issues Handled correctly?
 

I really like this idea. The full bracket would take some time, but the practice matches would be good pumping for the crowds and I would love to see all alliances and what they are capable of before the matches start.

Re: Einstein Field issues Handled correctly?
 

You are correct. The issues on Einstein were just more visible. Issues have been ongoing all season at multiple venues.

I'm not entirely sure that all the problems are firmware and hardware provided by FIRST. In some cases teams did have some influence over the issues. Not that teams didn't expend absolutely every effort to work out their issues when they had the tools to find them.

However, I am entirely sure that FIRST needs to work on the troubleshooting process. Not merely for Einstein but competition wide. FIRST needs tools to solve these problems or no matter the brilliance of everyone involved the deadlines will be a disaster.

Re: Einstein Field issues Handled correctly?
 

So like a complete exhibition match where teams can go in guns blazing? I would LOVE to watch that. Maybe even throw in a 2v2 of all of the #1 seeds. ACvs.GN

Re: Einstein Field issues Handled correctly?
 

This.

100% correct. These problems were occurring on other fields, at other competitions, and other years with this control system. Something needs to be done about the system, not the implementation. The system they have is the problem, not specifically how they went about handling Einstein (though that could have been improved too, it's not the root of the problem).

Re: Einstein Field issues Handled correctly?
 

I would like to start off by congratulating 180, 25 and 16 on their win and performance, it was an absolutely awesome alliance to behold.

Regarding the handling of Einstein...I felt like there may have been a better way to handle it, but I don't have a solution for how to do it better, other than to give more time to the field crew by running awards consecutively, and possibly not to crown a champion. I do understand, however, that there were very limited options. I understand it's supposed to be a "show", but honestly, proceeding with a tournament which is obviously flawed is a pretty bad show. Ultimately, however, there was no good solution. Everyone lost on Einstein because it was impossible to play fair matches. I hold the utmost respect and trust for the veteran crew that was running Einstein, and I believe they did everything in their power to fix it, but at that point things were moving too fast and the problem was too far out of their hands to reasonably do anything else. The decision was made to move on, and I know everyone on the field at the time understood the implications of that decision.

The problem with admitting that there is a serious issue that cannot be resolved is that it throws doubt on the entire season. These problems have been happening since week 1, and to admit on Einstein that the issues were not, in fact, able to be fixed by teams means that everyone who experience this issue at some point during the season would feel cheated.

The problem with proceeding with play is that people will forever question the legitimacy. There is no doubt in my mind that all 4 of the alliances on Einstein had a legitimate shot at the championship, and they all exhibited phenomenal play throughout their divisions. While the outcome may have been the same, I think everyone including the teams on the field will agree that Einstein was decided by the system issues, not by the alliances' play, and that is a real shame.

Ultimately, the solution is for FIRST not to ignore the problems like this when they first start cropping up, and make a concentrated effort to diagnose and resolve them as soon as possible, instead of proceeding with the status quo that anything which can't be diagnosed from the field must be a robot problem. This was not an isolated incident, it was a widespread problem. Enough so that I predicted on Thursday (and several people can quote me as such) that every single finals matchup played would have a dead robot during at least part of it. I don't know if it came entirely true, but the fact that I was right for Archimedes, Curie (I heard Newton had a dead 68 as well, but I didn't personally witness it) and Einstein means that something is very very wrong. That doesn't even count other dead robots during quarters and semis like 330 and 1717.

I would never EVER wish for this to happen to a team, but at the very least, FIRST is now recognizing that they need to take a much closer look at the system. I expect this problem will not happen next year.

The last point I want to make is similar to what Kevin Sevcik said earlier. This may or may not be a "field" problem, and it may or may not be a "robot" problem, but it is very definitely and obviously something which many many top teams and engineers have not been able to diagnose and do not have the ability to fix. There is a problem with the control system and it is something the teams do not currently have any way to prevent from happening, because nobody can tell them how to fix it. This, to me, is inexcusable. I don't care if FMS says everything is working fine, if the best teams in FRC are having that many problems and NOBODY can tell them why, you can't just tell the teams "tough, it's not our end so it has to be yours". I mean no slight to the FTA's, they were just following protocol and doing their best job to diagnose things on the fly. I mean in the sense that FIRST has not publicly expressed any concern or attempt to resolve the problems. If they had been, I really wish they had been public about it.

I know NI has been looking at reports of these issues throughout the season and trying to root out the problem, but I haven't heard what their theories are and if there's any known solution. Likewise, I don't know if their investigation was prompted internally after seeing it at events, or by FIRST.

TL;DR, FIRST needed to be more transparent and proactive about addressing the issues throughout the season. It was apparent to many people that there was a serious problem, but by the time it surfaced on Einstein, there was no good solution.

971 experience the same symptoms as exhibited on Einstein during several of our matches on Curie, and we were unable to diagnose the issue, just like everyone else. I'm not particularly familiar with the control system so I can't share many details, but we never determined the root cause of the problem. It may or may not be the same as what other teams experienced. Saying the issues was "on our end" is the same thing teams have been told all season. "Our end" doesn't mean "our fault" or that we had the ability to fix it. We worked in our quarterfinals matches, but we don't know why.

78 also died in F1, but I can't speak to the cause either.

Re: Einstein Field issues Handled correctly?
 

and you know if you compare the teams side by side with stats and hard evidence it is apparent who the real world champs should be.

Re: Einstein Field issues Handled correctly?
 

i mentioned in the other thread. This is a mission critical network during the competition. If I was in charge and my network engineering group came to me with failures and could not tell me why, it would be unacceptable. They need the tools and knowledge to track every statistic and frame/packet on the network. Like most engineering groups, they are probably underfunded, understaffed, and lacking tools to adequalty manage the network. Any dropped/delayed frame/packet should throw up a red alarm on a management system in under a second. All traffic should be mirrored to a 2nd port where it is saved. Using this saved file, an entire match can be run offline using a tool such as Wireshark. We also need a way to monitor traffic on the robot's bridge. If we know the root cause, we can fix it. There will always be exceptions which will have to be dealt with, but having he same thing take place more then a couple of times is avoidable. It may take a more exspensive device, but we need more visibility into the communication between devices. After this year, I think more teams will be debugging and monitoring during testing so they know what is "normal". It's another area that will benefit the students to learn. Everyone grab Wireshark and connect a PC with it to the DLINK and start playing:)

Congrats to the world champs!! Fantastic job!!

Re: Einstein Field issues Handled correctly?
 

Just to add a bit more information to the mix:
Although we certainly ceased to function in ways similar to the robots on Einstein during a couple of our matches, at least two of the matches in which we ceased to function were certainly caused by problems at our end (In Qualification 9, the joysticks didn't work, and in one other match, the main breaker tripped. We aren't entirely sure as to the cause of these problems, but they did not recur and they certainly weren't the fault of the field). As for all the other times we didn't function, we aren't sure why, but either it was the same as Einstein or it was a different problem. If it is the same problem as whatever happened on Einstein, then we know that we aren't alone in having the problem. If it is a problem with something on the robot, then that means that we may be able to debug it with our own resources. Just so long as FIRST is trying to find a fix to the problem, then I am satisfied. And given the email sent by FIRST earlier, it appears that I am satisfied :).

Re: Einstein Field issues Handled correctly?
 

Just in case people who are looking into the issue haven't already considered doing this, I think one thing I would certainly do is to set up 'tcpdump' to capture an entire match where the problem reproduces. In order to reproduce the problem, I'd consider using 'netem' to simulate an impaired network. What you really want to see happen at all times, but particularly under adverse conditions, is that each team gets an equal fraction of the available bandwidth, with the most important packets getting through and things like the video stream being throttled. It isn't too hard to make this sort of thing happen using TCP/IP. What you really don't want to happen is for any team to lose enough bandwidth that heartbeats get held off for long enough to cause problems.

We had very similar problems in 2010, when rookie teams had a different radio than veteran teams. This seemed to hit Red 2 especially hard, IIRC. I imagine the teams on Einstein all had this covered, but we have always tried to mount the radio in the open, high and not surrounded by metal, as well as put a glob of hot-melt glue so it is holding the power plug in place. If your team doesn't already do these things, they can only help.

Congrats to all, it was hard to watch things unfold as they did but it is a fantastic achievement to make it onto Einstein and no one should let what happened mar the pride in finishing in the top 12 teams in the world. The Chairman's Award was very well deserved and more than anything, FIRST really is a fantastic program that continually works to improve and everyone involved deserves to be recognized for inspiring everyone else and for demonstrating Gracious Professionalism under very intense circumstances.

Re: Einstein Field issues Handled correctly?
 

Nuttle - you are a CLI guy:) Your asking alot - learn UNIX and use a command line. Holy Cow, my heads going to explode:D

Good advice though - you don't have to spend any money to do some really cool stuff.

Re: Einstein Field issues Handled correctly?
 

Perhaps they should have been playing this remix instead of the FIRST Robotics version on Einstein...

http://gobarbra.com/hit/new-4e6fed9c...7128c25cdfe22f

My apologies to the teams, this was handled very poorly in my book and it is my opinion that you should all walk away feeling like champions.

Re: Einstein Field issues Handled correctly?
 

I agree! There were problems all season and I've heard reports of problems on all of the fields at CMP. However, the problems were worse on Einstein.

So what is different about Einstein that might explain things?

A few have been identified. More spectators means more texting, wifi, etc. The field was untested. The spectacle was greater with more lights, etc. The storm rolled in.

One wayt to troubleshoot is to take a list and add anything else that you can think of and then eliminate them one at a time. I'm sure each of us have our favorite theory of what was the cause.

Just to plug for mine, the problem appears to be intermittent, tends to affect teams with lots of comms back and forth to their staion, and tends to happen during elims when the fans are paying attention and using their cell phones and wifi tethering a lot. Therefore, I think it is radio interference from a combination of sources. If that is correct, then this is going to be very hard to pin down concretely, but realtively easy to fix. A different radio or frequency could fix the problem.

Re: Einstein Field issues Handled correctly?
 

My guess is that the issue is triggered by more bandwidth being needed than there is available. So, robots that upload lots of telemetry, all have video feeds, etc. combined with anything that causes less availible bandwidth is the magic combination. Fixing whaever is limiting the bandwidth makes the problem go away, but really doesn't solve the deeper issue. When bandwidth is tight, packets drop, TPC requests retransmissions, and this only results in more bandwidth being needed. Traffic on some critical connections gets held up for long enough that robots die and do not recover. If this is right, the old radios in 2010 turned up the same underlying issue but the problem went away when everyone was told to use the new radios.

Depending on how things are implemented, a single robot could have more than one video stream coming from a single camera -- one for the dashboard and another one for off-board target tracking. One way to simulate this would be to open extra web sessions with the camera. The 'netem' tool I mentioned before can simulate a network with various issues in the network and is a more controlled way to try to cause this type of problem.

This wouldn't be an issue when using a tether, and a single robot running over wireless would have six times the bandwidth as when a match is being played and so would not likely see this either. The exact traffic when running under the FMS might be different as well. Using tcpdump / wireshark would allow digging into the problem.

Just thought I'd throw this out since this angle hasn't come up in this thread and it is one theory that should be considered, I think. THe solution would involve taking steps to make sure the critical data always gets through by limiting the bandwidth that is allowed for less critical data when bandwidth is tight. Giving each team an equal amount of bandwidth is only fair, and things like the video feed will do OK with limited bandwidth, usually by dropping frames. It would be good to have a way for teams to test with limited bandwidth as well. It would be possible to have a gauge on the driver station showing bandwidth used or something along these lines to help teams catch issues they can contol that cause them to use more bandwidth than they need and generally be more aware of this.

Re: Einstein Field issues Handled correctly?
 

78 died due to the power connection to the wireless bridge coming out after an impact with the barrier shortly after hybrid mode.

Re: Einstein Field issues Handled correctly?
 

If you loose comms due to something on your robot unplugging then it is clearly your problem. Been there done that. BUT these field issues are embarrassing to FIRST or at least should be. I am not an engineer but if I remember right we are still on communications with Voyager 1 at something like a bazzillion miles from here and it is using a 5 watt transmitter. There has to be a better more robust more resilient way to make this work even if we have to go to a completely different control system.
I am not pointing fingers. I know it is a difficult situation to debug. It is just so frustrating for all involved. We were smacked by this in 2010 and it is still an issue. Curie had very few if any of these issues. What was so different there?

Bruce

Re: Einstein Field issues Handled correctly?
 

This concept DOES hold some water, especially in the context of Einstein.

It's reasonable to believe that the robots that reached Einstein would be making more effective use of the bandwidth available (sending streaming video back to the DS, etc).

It's then also reasonable to believe that if such bandwidth requirements are occupying near-to the full capacity of the link, and you start adding more of these bandwidth hungry robots to the network, problems occur due to dropping packets flooding the network with retransmits.

Maybe things could be improved by forcing camera information over a non-handshaked protocol like UDP, where no retransmission occurs if there are errors (after all, retransmitting an old camera image doesn't help, when there's a new image to transmit instead).

I agree that it would be really nice if HQ could get 6 of the 12, or all 12 Einstein robots, with the Einstein field, and test things out at HQ.

Even then, I'm not sure things would fail without the crowd of thousands of WiFi enabled devices inside a pseudo-faraday cage with it.

Re: Einstein Field issues Handled correctly?
 

I'm not an engineer. I cannot speak to how the field works, or how it didn't during Einstein.

However, your statement struck me. I was sitting down in the front during Einstein, as I was speaking during the Awards ceremony.
Those watching on the webcast probably didn't see this:

Dean was down on the floor, behind the scoring table, during every match... trying to help figure out what was going wrong and how to fix it. He was ushered back to the stage for each speaker, but he wanted to help fix it too.

He was upset, the staff was upset... we all were.

As you have seen in the letter from President Jon Dudas... FIRST is going to try to figure out what happened and make a solution. I fully believe that it's important to recognize each of the Division champion teams for their efforts on Einstein in some way... and then work on resolving the connection issues for next year.

I had no bearing on what happened, but I am truly sorry it did. I'm sure FIRST will make sure this never happens again.

Re: Einstein Field issues Handled correctly?
 

Whatever happened, has happened.

We as a community do not need to speculate on why it happened. We don't really even need to postulate on how to keep it from happening again. What we need to do is trust in the number of really great people who are working on it, and focus on moving forward in a positive way as a community.

How we as a community conduct ourselves now will define our collective character. Can we move forward in a constructive manner?

My personal thoughts are here:
http://blog.iamjvn.com/2012/04/react...ebrations.html

I had a great time at the FRC Championship, and I hope everyone else did too. Good experiences can happen even when things go badly. Let's make this into a good experience.

-John

Re: Einstein Field issues Handled correctly?
 

I just want to say thank-you for saying that.

The Volunteers and Staff who deal with this aspect of the competition (FTA, FTAA, CSA's, etc.) have a very strong passions for what they do. These guys really want to get every robot working on that field, and it hurts them too whenever a team runs into problems, such as what happened on Einstein.

I personally have put my full trust behind them. I have never doubted there calls, and I feel others should do the same.

So I agree, as a community we need to let these people do their jobs, and give them our full support and trust, so they can Improve FRC and give us the best possible experience the can.

Re: Einstein Field issues Handled correctly?
 

I think a lot of people needed to be reminded of just that.

Oh, and welcome back, JVN!

Re: Einstein Field issues Handled correctly?
 

Re: Einstein Field issues Handled correctly?
 

I believe that this will be remembered as a watershed year for FIRST

There are things that will change after this year.
There are many intelligent people cogitating on the season.
We will engineer a better solution.

This was a great game this year. The winning alliance were teams with 15, 16, and 17 years in FIRST.
The universe conspired and after all is said and one, the result was inspiring.

Re: Einstein Field issues Handled correctly?
 

Everyone has a right to be upset that it had to come to a failure on the biggest stage in FIRST, with the founder and his second in command becoming as distressed as the teams on the field, for networking problems that have persisted since the inception of the control system.

That doesn't mean you should question the integrity of the organization we support so fervently. I don't question that FIRST is looking into this seriously, I just wish this was made more pressing before now.

Re: Einstein Field issues Handled correctly?
 

Yes. We found that it was very difficult to triple balance when we never had had three moving robots at the same time... In all seriousness, we desperately wanted to triple balance but the communication issues our alliance experienced made this impossible. Both 1114 and 2056 were dead for significant periods of all three matches, while 4334 was motionless for the entire first match.

---

The issues of Einstein 2012 were highly unfortunate. There's no sugar coating it. For the rest of my life, I will never forget the looks on the faces of team members of every team down there. Not just the losing teams, but the winners as well. No one was happy about what happened. We all felt so helpless. Everyone was trying to console each other, but there was nothing anyone could say other than "it's not right".

The most striking moment for me was when 2056's six year alliance captain, Isaac Hunter, looked at me and said "Karthik, can't we do something about this? We can't just let this happen." I was left speechless. I'm used to being the person who always has something to say to make people feel better. But at that moment all I could conjure was "There's nothing we can do. It's over."

The silver lining of this incident was the enduring spirit of gracious professionalism between the 12 affected teams. Everyone was looking out for each other, the lines between alliances became completely blurred. In many ways it felt like there was one giant 12 team alliance that won Champs together. I hope that gets recognized in some way or another.

The statement put out by FIRST after the event was very powerful. I commend Jon Dudas for taking such swift action. I'm confident that he will lead a thorough investigation into the events of Saturday afternoon. I hope that FIRST will prove my answer to Isaac to be wrong. I hope there is something that can be done to rectify what happened. In fact, hope is the wrong word. I know and have complete faith that FIRST is going to fix this. Let's just give them time.

In the mean time, I'd like to congratulate our 2012 Champions of the World, Teams 180, 25 & 16. These teams should carry that title with both honour and pride.

Re: Einstein Field issues Handled correctly?
 

It was completely unacceptable

Re: Einstein Field issues Handled correctly?
 

I would just like to note that this thread started as "were the issues handled correctly" not "were the issues acceptable". I think everyone agrees that we would rather a working Einstein, and matches will all robots working. The question is, was there anything FIRST could have or would have done, given the situation?

Re: Einstein Field issues Handled correctly?
 

I believe that, on the day, FIRST (as far as we know) did what it could to fix the issues on Einstein. The real problem is, though, that these problems had been apparent for the past few days, weeks and even years, and FIRST and its top employees took every opportunity to blame communications issues on the teams until this issue made them finally realize that there was another problem. If they had actually addressed the issue earlier, this wouldn't have had to happen.

Re: Einstein Field issues Handled correctly?
 

Given the overall situation (extended history of problems), no, the issues weren't handled properly.

Given the immediate situation on Einstein... Tough call, but I have to go with yes, under the circumstances.

Overall, the "it's not the field it's your robot" type of statements--and the fact that much of the time, doing something to the robot made the situation better--served to not get the problem addressed sooner. Overall, that's an issue. Sweeping a problem under the rug doesn't make it go away. It's better to face it head-on and try to run some tests--I know at an internship after some testing of a troublesome part we were able to tell a customer, "When properly set up, the whatsits perform as advertised. We see they haven't been set up properly in your situation." (We were also looking at possible alternative methods to perform the function of the whatsits in case of inability to set up properly.) If you think there's an issue, go for the root cause.

For the immediate situation, there are a lot of factors going into this one.
First, you have the published documentation. Replays are by field fault or true tie only under the rules. And you can't for sure say that this is a field fault (we're assuming that you're looking at only Einstein). So, by ruling it a field fault and replaying the first two semis matches, when there was no concrete evidence that it was a field fault, they did do their best to accommodate teams. Positive point.
Second, you have the big picture. Looking beyond just Einstein, the robots worked better on the divisional fields. This puts the weight towards it being a field fault, so more replays should have happened. Negative points.
Third, you have the schedule. The Championship traditionally runs over--the only question is, how far? By issuing more replays, you lengthen the time--and shorten the night out--and possibly force teams to choose between staying or heading for home in time to get some sleep before Monday's return to the daily grind. This balances out the big picture look, so you do limited replays. Positive point.
Fourth, the weather. In bad weather, staying inside is the place to be--and you don't want to be running to vehicles in a hailstorm. But, it has the potential to interfere with all types of electromagnetic radiation, including the field control system wireless. The thing is, most electronics will be built to handle that interference these days, because it's a known potential issue. So that's a moot point (other than the keeping people inside because of weather part--a positive but unrelated point).

With the scale being balanced like that, it's time to play "benefit of the doubt" and "20/20 hindsight" cards and say that FIRST handled the immediate Einstein issues as best as they could under less-than-ideal circumstances, but needs to be more proactive in looking into possible root causes when untraceable issues crop up in large-ish numbers.

Re: Einstein Field issues Handled correctly?
 

Does anyone have a full field video that shows the scale of the issues?

From watching the web cast only I wasn't clear that everything I've read here was quite as bad as I've seen.

Re: Einstein Field issues Handled correctly?
 

The way in which those of you who were a part of the 12 team alliance are expressing how you dealt with and handled the pressure and frustration, is recognizing the qualities of champions in each of you. You are bringing this recognition to yourselves in ways that show true leadership and reflect the integrity of the experience. What happened was horrible. How the 12 teams responded was magnificent.

Jane

Re: Einstein Field issues Handled correctly?
 

It is not possible to change anything that has happened and under the circumstances everyone tried to make the correct decisions and behaved extremely well considering the drive and emotions present. We can't change what has happened but we can work together to understand the problem with facts and then design a solution. Those of us who have designed protocols and done in depth analysis of network traffic all know the performance is probabilistic and there are no guarantee performance metrics for this type of network. The cost advantage and ease of implementation for this hardware is significant and has to be balanced with the performance. There is no question that there are many hazards that exist in drive station software and also the CRIO basic design that were exposed by the level of network traffic that was present this year. There is a real difference between a design that works every time and one that works most of the time.
To solve this problem and move ahead we need to divide the problem into a couple of pieces.
First we need to find and remove the software hazards that make the driver station and the CRIO vulnerable to missing packet and possible lockup conditions. Some of these hazards may have been in the software for years but really were exposed by the level of network traffic this year.
Second we need to make a realistic model of the network traffic by in depth recording some trial events and looking at the options for network configuration that will improve the overall performance or issue clear limits that will bring the performance into an acceptable range. The quality of some of the wireless hardware may in fact play a major role in this analysis.
Let’s focus on finding solutions that after all it one of the life lessons FIRST is all about.

Re: Einstein Field issues Handled correctly?
 

Well said
Maybe we'll stop hearing how it is not the field and it must be our robot. The anxiety that Dean and Woody experienced was a small fraction of what the finalists and other teams have been experiencing.

Re: Einstein Field issues Handled correctly?
 

A quick disclaimer: I don't speak for the Robonauts, I speak for myself. Contents of this post are influenced by what I heard from my fellow mentors and students, but this is my opinion and you should not associate it with my team.

This is the real reason I am so upset. It's not just that we didn't get to play, it's that we didn't get to play and FIRST blamed it on us. "There was a problem in SF2-1; there may or may not have been a problem with SF1-1." That sentence, said by the emcee, seems like it was targeted directly at 118, at least to me. It sounds like they decided that we didn't deserve a second chance, but they were giving it to us only to appear fair. I was livid in the stands when I heard that, and I probably made my team look bad with some of my angry screams (for that, I apologize). But to claim it was our fault, after what happened in Connecticut and how hard we worked to replicate 364's issues at Lone Star, was just unbelievable.

Furthermore, the commentary during the matches made it sound as if we had control over how immobile our robot was. There's a reason we nicknamed Archimedes the Eh Team beyond them being Canadian; they were really incredible, and deserved to be thought of as the A team, the best FIRST could offer. But from the commentary of the match alone, you'd think Curie was playing against a bunch of rookies. It was very derogatory. There was nothing FIRST could do about the connection errors (at least, that late in the season) or the schedule or even the tornado. The show must go on. But where FIRST went wrong was making it seem like the teams were responsible, and then actively ignoring our plight during match commentary. It was rude, it was unprofessional, it was about as opposite as gracious as you can get. That's where FIRST abandoned its values, and that's what FIRST needs to correct.

We as a community will do everything we can to help FIRST get the communication issues resolved. We will overlook what happened on Einstein in the name of science, and continue to recruit and spread the values of FIRST. But I personally will not stop expressing my opinion in forums such as this until FIRST apologizes for the way they treated the 12 teams on Einstein. Kamen just knelt by the side of the field and looked upset while watching the train derail, and then pretended there wasn't a problem when it came time to pass out awards. You cannot tell me that was the right thing to do. FIRST has never needed accountability before, but they do now. 16, 25, and 180 deserved the win, but the rest of the alliances deserved to play.

Re: Einstein Field issues Handled correctly?
 

I'm really hoping that because these particular radio issues that affected Einstein were not only so debilitating but also so public (biggest matches of the season and all) that it will really, well for lack of a better word; "shame" FIRST into finaly looking seriously at the issue of connectivity. For as long back as I can remember (before they switched to routers) there have been connectivity issues. Years and years now this has been going on, affecting one team or the other with some field set-ups exacerbating the issue further.

It's just been a running joke, like the old submission process for Autodesk Inventor files (Pack & Go assembly - upload files - files fail to upload - contact Autodesk - Autodesk responds and says since the uploads failed for everyone they'd push the deadline back) until they changed the submission process for this season.

So, again. Here's hoping they take a very serious look at the issue. As far as some of the opinions expressed in this thread; I do think the fault lies somewhere with the consumer routers they're using and I do not think this is a "tainted" win. Every team had to deal with these connectivity issues. They never should have been an issue in the first place, but it happened and there's nothing we can change about that now.

My thoughts on Einstein 2012.
 

This could be a very long post, I will do the best to explain my full thoughts on the Einstein field issues.

Being the driver for 4334, I was front and center in experiencing this awful event. I can't tell you how awful it is for your whole alliance to go down during the championship finals. Forgive any bitterness I may have in this post, I am pretty biased on this event.

While I understand that FIRST had their reasons for handling things in the way that they did, it was completely disgusting. The volunteers that I dealt with were all less than helpful, they brushed me off more than once, and not once did they give us a straight answer. I realize and recognize that they didn't have answers, but the least they could do is admit there was a problem (which they did when we replayed those matches). It made it hard for anyone to believe that they fixed the problem when the problem continued. I was standing there listening as a volunteer tried to tell 2056 is was a battery issue with their bot. I mean, 2056 has had a perfect record this season in terms of up time and robot reliability. Loose connection in one match; improbable but plausible. Loose connection in 2 matches; highly unlikely but possible. Loose connection in all three matches; I mean, look at the stats and tell me that's what happened.

They did the right thing by replaying the first two matches - they did the wrong thing by powering through and denying that there was a problem. I don't think I'll ever forget the moment I lost faith in the FIRST system - just the injustice of what happened was enough to make me want to quit FRC. After a match when so many robots went down, holding faith that they would replay, and then not doing it... It's ridiculous. If there was a massive hole in the ice at the Stanley Cup finals, they'd stop and fix the ice. This was a world championship event, and the field didn't work? That should be the least of anyone's concern. And to think - if I won worlds, went home, got on Delphi, and saw that there was problems with the field, I would seriously question the legitimacy of the award. That alone should have been reason enough to fix the field. And think of how bad this made FIRST look - people who don't really know anything about the dynamics of Rebound Rumble and the structure of the FIRST ranking system/elimination processes must be thinking "Wow. How bad were the rest of the robots?"

As anyone involved in the community knows, FIRST is all about gracious professionalism, honesty, integrity... And they informed 10,000 kids by example that none of that matters when your reputation is on the line.

Now, I'm not going to go into "well, this would have happened if.." The All-Canadian Alliance has been the FIRST Canada dream for years now, and this year it became a reality. People recognized how strong an alliance we were. We had a really good run, and it stung to see and feel it end how it did. Anyone who's ever gone down during a match can agree with me it's probably the worst thing that can happen to you. It's horrible. And then for them to turn around and tell us it was our robots? Add insult to injury, why don't you.

Honestly, I'm impressed by FIRST's acknowledgement of the problem. I think we'll probably never know what caused it. I've heard all kinds of ideas on what happened, from thunderstorm interference to conspiracy theories involving the War of 1812. (I actually overheard someone say that last time there was a bunch of Canadians in the White House they burned it down).

So, as respectfully as I can, and with as much gracious professionalism as possible, FIRST screwed up big time. I hope they took away from this a valuable lesson.

I guess above all, I wanted to show the world our siiick triple balance. I think we can all agree it was a thing of beauty.

We need to make sure something like this never happens again. It's unprofessional, low, and unbecoming.

Learn from the past and apply it to the future.

On a lighter note, the rest of the championship was amazing. So good to meet so many amazing people, I had a great time aside from the last night.

Mac
ATA 4334

Re: Einstein Field issues Handled correctly?
 

In declaring a winner, you're also inherently declaring losers. How was it fair for FIRST to tell 118, 2194, 548, 1114, 2056, 4334, 233, 207 and 987 that they lost? None of these teams ever lost a legitimate round at the championships. This is the biggest question mark for me.

I realize I'm treading on thin ice for saying what I'm about to say and I want to preface it by saying that I have the utmost respect for 25, 180, and 16. They are very classy and professional teams, with incredible robots, and they all deserved to win a championship title. The problem is, the 118, 2194, 548, 1114, 2056, 4334, 233, 207 and 987 are all deserving as well. Nobody won or lost a legitimate match on Einstein, and I question FIRST's decision to award a title at all.

I really want to stress that in saying this I mean nothing against the winning teams. Given the decisions FIRST made, they followed exactly what they were supposed to do. They were even gracious and professional enough to hold back on celebrations out of respect for the other teams there. They were truly a class act. I want to also stress what I said in the beginning; FIRST inherently told the other 9 teams on Einstein that they had lost in declaring that one of the teams had won, and I don't believe this was fair to given the way the matches were played.

And I feel for the losing teams, but I feel even worse for the winning teams. They were put in a difficult situation, and they handled as well as anyone possibly could. They all worked so hard to win a title at the championships, and they deserving achieved their goal, but were unable to celebrate the accomplishment to the fullest. That must be an awful feeling, to know you won, and know you deserved to win, but have some form of an asterisk next to the title. I feel bad for them. They are great teams, and I wish them the best of luck next year!

Re: Einstein Field issues Handled correctly?
 

I think we all feel the same way, some more vocally than others. Our alliance felt reason to celebrate winning Curie, beating a very good alliance of Daisy, Poofs, and Air Strike. We did NOT feel so good about beating the Maple Leaf Alliance, because those were not fair matches.

At this point, we only need to trust that by next year, there will be a reliable control system, whatever it takes.

Re: Einstein Field issues Handled correctly?
 

It seems to me like the teams on Einstein understand the spirit of FIRST more than the people that are running FIRST. All season these problems have persisted and teams were told it was their own fault and were robbed not only of the time they put into FRC but also the money they invested in the competition. The idea that it needed to happen on Einstein before something was done is truly disgusting to me. Today for the first time in my 7 years in FIRST I am not proud to consider myself a part of this great organization. Then again I am proud to associate myself with the teams that played on Einstein and acted the right way. FIRST should take a lesson from these teams, a lesson they were supposed to be teaching these teams not the other way around.

This is just my personal outrage because I have been waiting since week 1 to see how this game would play out on Einstein and now I never will.

Re: My thoughts on Einstein 2012.
 

I've seen this over and over.

I've commented on it here (see post 204):
http://www.chiefdelphi.com/forums/sh...104713&page=14

The problem is that if it is power quality issues they suspect, the field folk lack the tools themselves to confirm promptly and to some extent so do you.

So it's an easy fix all to move you forward. They tell you that it's some part of the power system. They have some authority so you naturally move forward.

Course when you realize that you've exhausted all your capability, you have not satisfied them, and you're still stuck. Then you start to get the impression they are messing with you.

I've not seen anyone recently make such a recommendation just to cause you problems to a team. However, if they did you'd never get past it at a competition....no matter how hard you try at the average competition I've seen you'll never prove your point with the tools on hand.

Sometimes I think that the field personnel when increasingly confronted over the issue naturally side with the familiar and assume that it's more likely it's your problem then the field. Their observation is just as anecdotal. You could have dropped a DC-DC converter after 3 matches as we did. Not a wiring problem in sight, but a bad D-Link AP power supply none-the-less.

In any case, neither the field folk or the teams should point fingers at each other. More often than not you didn't have the tools to remove the doubt, that doesn't make the concern invalid.

Re: Einstein Field issues Handled correctly?
 

There was discussion while matches were being played on Einstein about giving everyone there a finalist medal. Most people there were instantly turned off to the idea since that meant there would be no winner.

Honestly, to me, that would've been more offensive than anything. I'd rather lose to the field and have a winning Alliance of 16, 25 and 180 than be called an Einstein Finalist along with 11 other teams.

Re: My thoughts on Einstein 2012.
 

Thing is, we know it wasn't the robots. After every match we took them backstage and ran a full systems check, and they were all fine. And the bots weren't always down for the whole match. Just sometimes. They kept coming online and going offline without rhyme nor reason. We know it was not our robots.

Re: Einstein Field issues Handled correctly?
 

Perhaps everybody should have been given a "champion" medal, trophy, and banner.

Re: My thoughts on Einstein 2012.
 

For argument's sake, let's say hypothetically that is what happened to 2056's bot. How do you explain the same thing happening to all the other robots that went down during the finals? I kind of don't think that all the robots had the exact same electrical problem.

Re: Einstein Field issues Handled correctly?
 

Well, seeing as there was no legitimate wins or losses on Einstein, it would stand to reason that everyone tied. However, I'm not sure if that would be more embarrassing for the teams or for FIRST. There is really no happy medium. Plus what would they do about visiting the White House and all that?

No matter how you cut it, it's the same garbage. The only thing we can do it figure out what went wrong and try to prevent it from ever happening again.

Re: Einstein Field issues Handled correctly?
 

AFAIK that is a CCW thing... so 1114 gets the honor this year.

Re: Einstein Field issues Handled correctly?
 

But they're Canadian, so shouldn't they visit Canada's prime minister or something?

Re: Einstein Field issues Handled correctly?
 

What color is his house? :rolleyes:

Googling last year, it looked like one student from each of the 3 winners went.

Re: Einstein Field issues Handled correctly?
 

I believe they handle it on a year-to-year basis. Some years it hasn't happened, and the content of the invitee list has varied.

I'd rate it very unlikely that the White House would invite 1114. Rideau Hall, maybe. (David Johnston used to be the University of Waterloo's president, and so is probably acquainted with FRC, via the Waterloo Regional.)

Re: Einstein Field issues Handled correctly?
 

Makes sense.

And meeting Stephen Harper isn't really that impressive, quite honestly. He has a house in Calgary and goes to the movies and stuff with his family sometimes. Meeting him's really nothing to write home about.

It would stand to reason that his house would be red and white, however that is not the case.

I kind of think that this is a little off-topic now.

Re: Einstein Field issues Handled correctly?
 

At post #194 we'll cut you some slack; it seems to make it's way back on topic every now and then.

Re: Einstein Field issues Handled correctly?
 

Robotics is the greatest organization in the world. I can't even imagine that FIRST will continue to leave the Einstein teams disillusioned, disappointed, &/or with the myriad of negative feelings they may currently possess. As Dean Kamen began to speak on Saturday, the crowds went wild! I said to my family, "Dean is the Rock star of Robotics!" I thought I was about to watch one of the greatest moments in my son's life, instead I witnessed his most disappointing moment to date. It's not about winning or losing, it's how we played the game, but it felt to me that Team 118, The Robonauts,(as well as the other Einstein competitors) didn't even get to play. I so wish we could recapture the initial excitement and redo the Einstein competition less the atrocities that played out. But short of Dean inventing a time machine, we cannot.
I challenge FIRST to some homework. Find the problem and a solution so no team in the future ever has to deal with the horror that these teams are enduring. This hasn't ended as many replay Saturday over and over in their head & continue to relive the event. They need adequate closure. Which leads me to another homework assignment: find a way to make it up to these teams. These teams have shown their gracious professionalism in an unbelievable manner and I have no doubt FIRST will do the same. An organization that stands for gracious professional won't possibly disappoint these phenomenal teams again....will they?

Re: Einstein Field issues Handled correctly?
 

I know this is totally off-topic, but that was one of the first things to pop into my head after "Finally!" and "YES!" when I heard 1114 won. :P:o