Einstein Field issues Handled correctly?

[Chi Meson]

Originally Posted by Kris Verdeyen

Don't have anything to add except to say that this is not true. The problems in CT were never diagnosed. We swapped out a cRio in Houston because we saw the same thing in a practice match, and couldn't bear the thought of doing nothing, even though the cRio swap didn't make much sense. We had no problems after that swap until what you saw on Einstein.

I'm glad to hear you state this. This is what I had heard through the channels, but I had yet to hear it firsthand. I recall the announcement at CT, that there was "no way that the field could have caused a robot's cRIO to reboot." I subsequently talked to about a dozen engineers who said (while never saying "that's not true") words to the effect "there's no way you can make such a blanket statement as that."

I'm split when it comes to answering the OP question. There is not much else the field administrators could have done. To put it as simply as possible, there IS an issue. The issue has affected teams that have the BEST software and engineering minds. The problem/s remain elusive and diagnoses have been numerous and varied.

Administration needed to make a decision, and they did not have all day to do so. It looked like a no-win decision.

Robonauts, we all ached for you. We hope we get to see you triple-balance at least one more time!

Let's not forget to congratulate the champions.

[sgreco]

I would blame the preparation more than anything. There's no excuse not to have the field ready for FRC use. I'm more curious than anything to see what FIRST has to say: what the problem was; if they diagnosed it; why they didn't move to another field; why they replayed the first few matches, but then one after that; if the weather impacted their decisions; etc.

I'll give them the benefit of the doubt until I hear an explanation, but I really hope it's a good one. I think FIRST owes a good explanation to all the teams, volunteers and spectators.

[JJackson]

Originally Posted by Coach Tom

Was the same system used on Einstein last year (Logomotion)? If I remember correctly, during the last two matches of Logomotion a RED Alliance robot worked in autonomous and then did not work at all during driver control. Was this a robot problem?

We linked this problem back to the case shorting BB-775 that we were using for our arm. However last year during qualifications on Archimedes there was a blue driver station that was giving teams communications problems throughout the day ... us included.

[MarkoRamius1086]

I believe we need to go back to the videos, the full and unclipped videos, if such things exist, and look and see exactly who had issues and when they had issues. I was not there and the webcasts were focusing only on the robots that were functioning, so attempting to see who was not functioning was... near impossible.

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division? I have been hearing of definite issues with something on the 118 scene, and slight hints at something with 1114.

Bryan Culver:
While this problem has been brought to light by the Einstein matches I do not think it is new. At MSC 33 died in the semis about 3 seconds into teleop. In the finals 67 had similar looking drop outs although I cannot confirm that it was in fact an identical problem. 1114 also seemed to have issues although those could be on the robot side (Both 67 and 1114 were connection issues.) Perhaps we, as a community, should begin documenting all of these failures throughout the season in order to identify patterns and the actual quantity of failures. It is easier to accept there is a problem when you have data to back it up and with many examples of failure to look through it would hopefully make identifing the problem that much easier.
Regards, Bryan

Absolutely true and beautifully said. The first step is who, what, and where. The details, not the speculation.

[rainbowdash]

Originally Posted by Kevin Sevcik

This.

Yes, the issues on Einstein were horrible, but after the first replay all you could really do is watch the train crash happen. The single most important thing is how FIRST addresses the problem in the coming weeks and months. If there isn't any comment or statement out of FIRST within a week or two, then I think we should all be highly concerned.

Originally Posted by SuperNerd256

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

Honestly, I would rather compromise time for a better, more fair matches.

[JesseK]

Here's a minor suggestion that may help test Einstein in the future:

Teams that are eliminated early in QF's from their respective divisions could go do a 'test run' match with some stipulations (stay off the bridges so they stay shiny, the balls may be horribly scuffed, etc). That way Einstein field can be tested with 32 different setups, including those with camera traffic, custom UDP sockets, and "nothing but dashboard" screens.

[Gregor]

Originally Posted by MarkoRamius1086

And partially related to that note, was there a strategic reason that "Team Canada" was not doing the triple balance like they had done so many times reliably and flawlessly in their division? I have been hearing of definite issues with something on the 118 scene, and slight hints at something with 1114.

It appeared that during SF 2-1 or 2-2, each robot on the 2056 lead alliance had some time when they were unable to operate, whilst each robot was able to move at other points in the same match.

[Dad1279]

Originally Posted by SuperNerd256

Honestly, While I completely disagree with how things were handled, what else could they do? All other fields were torn down, and they spent every second not playing trying to find the root of the problem.

They could have run a match swapping red & blue. That may have helped debug, and identify if it was a field or robot issue. They could have also asked that the teams involved ship their robots back to HQ instead of home, so the engineers could try to recreate the failure and set up the Einstein hardware with these robots again.

Now that the field is broken down, and the robots that were involved are gone, all that they can do is look at logs and wonder if they have the pertinent data.

[Duke461]

Not trying to be condescending here or anything, but I didn't see any representative of any team so far post what they personally tried to do to remedy it.

During one of my team's matches, we were motionless, but after we restarted the cRIO during the match, we got comm's and code and were perfectly fine.

Did any of these Einstein teams (esp. 118) restart/reboot their cRIO during the match?

-Duke

[Culvan Van Li]

I think they did the best they could in the time they had.

I would've liked to have seen them move to matches on Einstein sooner. That could've given them more time to troubleshoot the problem by putting more of the speeches and awards in between matches.

I'm amazed that this seemed to be a mystery. I work on telecommunications equipment. Part of my job is analyzing logs of communications failures. Those logs will usually pinpoint problems pretty well. Perhaps that's just because the FCC has set the bar so high. Compared to a worldwide telecommunications network the setup for the field shouldn't be that complicated. It seems to me that FIRST probably needs to add more troubleshooting and monitoring capabilities. The good news is that they control the software in the field, they control enough of the software in the robot, and they control enough of the software in the driver's station to do quite a bit. I think they probably need to add some more capabilities inside the robot, like have it log the battery voltage for the entire match inside the cRio. Ideally the system would be able to detect common robot failures that are completely unrelated to the field. That could really help improve the quality of the "it's not the field" message given to teams. I think we all want to see the best quality matches. I wish I could help more.

Andy

[steverk]

Originally Posted by efoote868

This was what I was thinking of:
http://en.wikipedia.org/wiki/U-NII
Where radar avoidance is mentioned several times.

Good thought, but the Robonauts practice shop is adjacent to a major airport.

Further, they are also within a few miles of the NWS office in League City Texas, which has a doppler radar. That's got to be closer than the radar that covers the Edward Jones Dome.

Therefore, I think it is unlikely that they would not experience problems in their practice building, but would at the dome.

Does anybody really think the Robonauts would allow such problems to go unresolved if they could duplicate them in their shop?

[Holtzman]

Originally Posted by Duke461

Not trying to be condescending here or anything, but I didn't see any representative of any team so far post what they personally tried to do to remedy it.

During one of my team's matches, we were motionless, but after we restarted the cRIO during the match, we got comm's and code and were perfectly fine.

Did any of these Einstein teams (esp. 118) restart/reboot their cRIO during the match?

-Duke

How exactly would we go about rebooting our crio mid match when we have no communication with our robots? Walk out on the field and cycle the main breaker?

We will share our full thoughts and experiences in the coming days.

[steverk]

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.

[Brandon_L]

Originally Posted by Deetman

were not caused by general electromagnetic interference with the WiFi system. I'm not discounting multiple WiFi networks being present confusing the routers on board our robots, but that is not directly related.

I may be a bit late on this (I only read up to like page 4 so far, I've been on a 15 hour bus ride back.) but I did check the wifi signals in the dome while Einstein was going on. I only picked up about 4 signals, only like one or two were somewhat strong.

Also one question that arose on the ride back from a few people: Why would the teams agree to play on a broken field?

[RyanN]

Originally Posted by steverk

Has anybody stopped to consider the lighting on Eistein?

The lighting was very different on Einstein than it was on any other field. It was much more involved. It had moving spotlights. It was routinely dimmed.

Theatre lighting is controlled by a several pieces of equipment called "dimmer packs." In recent years, they went to wireless controls on the dimmer packs, so ugly or lengthy control wires would not need to be strung to the lighting racks.

To quote wikipedia directly:

Recently, wireless DMX512 adapters have become popular, especially in architectural lighting installations where cable lengths can be prohibitively long. Such networks typically employ a wireless transmitter at the controller, with strategically placed receivers near the fixtures to convert the wireless signal back to conventional DMX512 wired network signals...The first commercially marketed wireless DMX512 system was based on frequency-hopping spread spectrum (FHSS) technology using commercial wireless modems.[8] Somewhat later some venders used WiFi/WLAN technology. Other later generation systems still used frequency-hopping spread spectrum (FHSS) technology, but at higher bandwidth. FHSS systems tend to disturb other types of wireless communication systems such as WiFi/WLAN. This has been solved in newer wireless DMX systems by using adaptive frequency hopping and cognitive coexistence, a technique to detect and avoid surrounding wireless systems, to avoid transmitting on occupied frequencies.[9]

(http://en.wikipedia.org/wiki/DMX512)

So could this be part of the problem? Was the transmitter near the wireless access point? Hopefully someone familiar with the field can answer some of these questions.

I was sort of thinking about that earlier...

Back when I was in high school, I was giving a demo at a basketball game. The score board was programmed wirelessly from a table on the side of the court.

When I drove the robot in front of the table, the robot died... I had to push it out of the area of that table, and it regained control.

This was using the IFI system, on the 900MHz radio, but it seems like everyone is moving their stuff to 2.4GHz nowadays.

Good thought!