|
|
|
![]() |
|
|||||||
|
||||||||
| View Poll Results: What do you think? | |||
| They handled it correctaly |
|
51 | 12.81% |
| They did not handle it correctly |
|
114 | 28.64% |
| It was horrible |
|
220 | 55.28% |
| Other post below |
|
13 | 3.27% |
| Voters: 398. You may not vote on this poll | |||
![]() |
| Thread Tools |
Rating:
|
Display Modes |
|
#151
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
Quote:
|
|
#152
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
Quote:
100% correct. These problems were occurring on other fields, at other competitions, and other years with this control system. Something needs to be done about the system, not the implementation. The system they have is the problem, not specifically how they went about handling Einstein (though that could have been improved too, it's not the root of the problem). |
|
#153
|
|||||
|
|||||
|
Re: Einstein Field issues Handled correctly?
I would like to start off by congratulating 180, 25 and 16 on their win and performance, it was an absolutely awesome alliance to behold.
Regarding the handling of Einstein...I felt like there may have been a better way to handle it, but I don't have a solution for how to do it better, other than to give more time to the field crew by running awards consecutively, and possibly not to crown a champion. I do understand, however, that there were very limited options. I understand it's supposed to be a "show", but honestly, proceeding with a tournament which is obviously flawed is a pretty bad show. Ultimately, however, there was no good solution. Everyone lost on Einstein because it was impossible to play fair matches. I hold the utmost respect and trust for the veteran crew that was running Einstein, and I believe they did everything in their power to fix it, but at that point things were moving too fast and the problem was too far out of their hands to reasonably do anything else. The decision was made to move on, and I know everyone on the field at the time understood the implications of that decision. The problem with admitting that there is a serious issue that cannot be resolved is that it throws doubt on the entire season. These problems have been happening since week 1, and to admit on Einstein that the issues were not, in fact, able to be fixed by teams means that everyone who experience this issue at some point during the season would feel cheated. The problem with proceeding with play is that people will forever question the legitimacy. There is no doubt in my mind that all 4 of the alliances on Einstein had a legitimate shot at the championship, and they all exhibited phenomenal play throughout their divisions. While the outcome may have been the same, I think everyone including the teams on the field will agree that Einstein was decided by the system issues, not by the alliances' play, and that is a real shame. Ultimately, the solution is for FIRST not to ignore the problems like this when they first start cropping up, and make a concentrated effort to diagnose and resolve them as soon as possible, instead of proceeding with the status quo that anything which can't be diagnosed from the field must be a robot problem. This was not an isolated incident, it was a widespread problem. Enough so that I predicted on Thursday (and several people can quote me as such) that every single finals matchup played would have a dead robot during at least part of it. I don't know if it came entirely true, but the fact that I was right for Archimedes, Curie (I heard Newton had a dead 68 as well, but I didn't personally witness it) and Einstein means that something is very very wrong. That doesn't even count other dead robots during quarters and semis like 330 and 1717. I would never EVER wish for this to happen to a team, but at the very least, FIRST is now recognizing that they need to take a much closer look at the system. I expect this problem will not happen next year. The last point I want to make is similar to what Kevin Sevcik said earlier. This may or may not be a "field" problem, and it may or may not be a "robot" problem, but it is very definitely and obviously something which many many top teams and engineers have not been able to diagnose and do not have the ability to fix. There is a problem with the control system and it is something the teams do not currently have any way to prevent from happening, because nobody can tell them how to fix it. This, to me, is inexcusable. I don't care if FMS says everything is working fine, if the best teams in FRC are having that many problems and NOBODY can tell them why, you can't just tell the teams "tough, it's not our end so it has to be yours". I mean no slight to the FTA's, they were just following protocol and doing their best job to diagnose things on the fly. I mean in the sense that FIRST has not publicly expressed any concern or attempt to resolve the problems. If they had been, I really wish they had been public about it. I know NI has been looking at reports of these issues throughout the season and trying to root out the problem, but I haven't heard what their theories are and if there's any known solution. Likewise, I don't know if their investigation was prompted internally after seeing it at events, or by FIRST. TL;DR, FIRST needed to be more transparent and proactive about addressing the issues throughout the season. It was apparent to many people that there was a serious problem, but by the time it surfaced on Einstein, there was no good solution. Quote:
Quote:
78 also died in F1, but I can't speak to the cause either. Last edited by Nuttyman54 : 29-04-2012 at 23:33. |
|
#154
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
Quote:
|
|
#155
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
i mentioned in the other thread. This is a mission critical network during the competition. If I was in charge and my network engineering group came to me with failures and could not tell me why, it would be unacceptable. They need the tools and knowledge to track every statistic and frame/packet on the network. Like most engineering groups, they are probably underfunded, understaffed, and lacking tools to adequalty manage the network. Any dropped/delayed frame/packet should throw up a red alarm on a management system in under a second. All traffic should be mirrored to a 2nd port where it is saved. Using this saved file, an entire match can be run offline using a tool such as Wireshark. We also need a way to monitor traffic on the robot's bridge. If we know the root cause, we can fix it. There will always be exceptions which will have to be dealt with, but having he same thing take place more then a couple of times is avoidable. It may take a more exspensive device, but we need more visibility into the communication between devices. After this year, I think more teams will be debugging and monitoring during testing so they know what is "normal". It's another area that will benefit the students to learn. Everyone grab Wireshark and connect a PC with it to the DLINK and start playing
![]() Congrats to the world champs!! Fantastic job!! |
|
#156
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
Quote:
Although we certainly ceased to function in ways similar to the robots on Einstein during a couple of our matches, at least two of the matches in which we ceased to function were certainly caused by problems at our end (In Qualification 9, the joysticks didn't work, and in one other match, the main breaker tripped. We aren't entirely sure as to the cause of these problems, but they did not recur and they certainly weren't the fault of the field). As for all the other times we didn't function, we aren't sure why, but either it was the same as Einstein or it was a different problem. If it is the same problem as whatever happened on Einstein, then we know that we aren't alone in having the problem. If it is a problem with something on the robot, then that means that we may be able to debug it with our own resources. Just so long as FIRST is trying to find a fix to the problem, then I am satisfied. And given the email sent by FIRST earlier, it appears that I am satisfied . |
|
#157
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
Just in case people who are looking into the issue haven't already considered doing this, I think one thing I would certainly do is to set up 'tcpdump' to capture an entire match where the problem reproduces. In order to reproduce the problem, I'd consider using 'netem' to simulate an impaired network. What you really want to see happen at all times, but particularly under adverse conditions, is that each team gets an equal fraction of the available bandwidth, with the most important packets getting through and things like the video stream being throttled. It isn't too hard to make this sort of thing happen using TCP/IP. What you really don't want to happen is for any team to lose enough bandwidth that heartbeats get held off for long enough to cause problems.
We had very similar problems in 2010, when rookie teams had a different radio than veteran teams. This seemed to hit Red 2 especially hard, IIRC. I imagine the teams on Einstein all had this covered, but we have always tried to mount the radio in the open, high and not surrounded by metal, as well as put a glob of hot-melt glue so it is holding the power plug in place. If your team doesn't already do these things, they can only help. Congrats to all, it was hard to watch things unfold as they did but it is a fantastic achievement to make it onto Einstein and no one should let what happened mar the pride in finishing in the top 12 teams in the world. The Chairman's Award was very well deserved and more than anything, FIRST really is a fantastic program that continually works to improve and everyone involved deserves to be recognized for inspiring everyone else and for demonstrating Gracious Professionalism under very intense circumstances. |
|
#158
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
Nuttle - you are a CLI guy
Your asking alot - learn UNIX and use a command line. Holy Cow, my heads going to explode Good advice though - you don't have to spend any money to do some really cool stuff. |
|
#159
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
Perhaps they should have been playing this remix instead of the FIRST Robotics version on Einstein...
http://gobarbra.com/hit/new-4e6fed9c...7128c25cdfe22f My apologies to the teams, this was handled very poorly in my book and it is my opinion that you should all walk away feeling like champions. |
|
#160
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
Quote:
So what is different about Einstein that might explain things? A few have been identified. More spectators means more texting, wifi, etc. The field was untested. The spectacle was greater with more lights, etc. The storm rolled in. One wayt to troubleshoot is to take a list and add anything else that you can think of and then eliminate them one at a time. I'm sure each of us have our favorite theory of what was the cause. Just to plug for mine, the problem appears to be intermittent, tends to affect teams with lots of comms back and forth to their staion, and tends to happen during elims when the fans are paying attention and using their cell phones and wifi tethering a lot. Therefore, I think it is radio interference from a combination of sources. If that is correct, then this is going to be very hard to pin down concretely, but realtively easy to fix. A different radio or frequency could fix the problem. |
|
#161
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
My guess is that the issue is triggered by more bandwidth being needed than there is available. So, robots that upload lots of telemetry, all have video feeds, etc. combined with anything that causes less availible bandwidth is the magic combination. Fixing whaever is limiting the bandwidth makes the problem go away, but really doesn't solve the deeper issue. When bandwidth is tight, packets drop, TPC requests retransmissions, and this only results in more bandwidth being needed. Traffic on some critical connections gets held up for long enough that robots die and do not recover. If this is right, the old radios in 2010 turned up the same underlying issue but the problem went away when everyone was told to use the new radios.
Depending on how things are implemented, a single robot could have more than one video stream coming from a single camera -- one for the dashboard and another one for off-board target tracking. One way to simulate this would be to open extra web sessions with the camera. The 'netem' tool I mentioned before can simulate a network with various issues in the network and is a more controlled way to try to cause this type of problem. This wouldn't be an issue when using a tether, and a single robot running over wireless would have six times the bandwidth as when a match is being played and so would not likely see this either. The exact traffic when running under the FMS might be different as well. Using tcpdump / wireshark would allow digging into the problem. Just thought I'd throw this out since this angle hasn't come up in this thread and it is one theory that should be considered, I think. THe solution would involve taking steps to make sure the critical data always gets through by limiting the bandwidth that is allowed for less critical data when bandwidth is tight. Giving each team an equal amount of bandwidth is only fair, and things like the video feed will do OK with limited bandwidth, usually by dropping frames. It would be good to have a way for teams to test with limited bandwidth as well. It would be possible to have a gauge on the driver station showing bandwidth used or something along these lines to help teams catch issues they can contol that cause them to use more bandwidth than they need and generally be more aware of this. |
|
#162
|
|||||
|
|||||
|
Re: Einstein Field issues Handled correctly?
78 died due to the power connection to the wireless bridge coming out after an impact with the barrier shortly after hybrid mode.
|
|
#163
|
||||
|
||||
|
Re: Einstein Field issues Handled correctly?
If you loose comms due to something on your robot unplugging then it is clearly your problem. Been there done that. BUT these field issues are embarrassing to FIRST or at least should be. I am not an engineer but if I remember right we are still on communications with Voyager 1 at something like a bazzillion miles from here and it is using a 5 watt transmitter. There has to be a better more robust more resilient way to make this work even if we have to go to a completely different control system.
I am not pointing fingers. I know it is a difficult situation to debug. It is just so frustrating for all involved. We were smacked by this in 2010 and it is still an issue. Curie had very few if any of these issues. What was so different there? Bruce |
|
#164
|
|||
|
|||
|
Re: Einstein Field issues Handled correctly?
Quote:
It's reasonable to believe that the robots that reached Einstein would be making more effective use of the bandwidth available (sending streaming video back to the DS, etc). It's then also reasonable to believe that if such bandwidth requirements are occupying near-to the full capacity of the link, and you start adding more of these bandwidth hungry robots to the network, problems occur due to dropping packets flooding the network with retransmits. Maybe things could be improved by forcing camera information over a non-handshaked protocol like UDP, where no retransmission occurs if there are errors (after all, retransmitting an old camera image doesn't help, when there's a new image to transmit instead). I agree that it would be really nice if HQ could get 6 of the 12, or all 12 Einstein robots, with the Einstein field, and test things out at HQ. Even then, I'm not sure things would fail without the crowd of thousands of WiFi enabled devices inside a pseudo-faraday cage with it. |
|
#165
|
|||||
|
|||||
|
Re: Einstein Field issues Handled correctly?
Quote:
However, your statement struck me. I was sitting down in the front during Einstein, as I was speaking during the Awards ceremony. Those watching on the webcast probably didn't see this: Dean was down on the floor, behind the scoring table, during every match... trying to help figure out what was going wrong and how to fix it. He was ushered back to the stage for each speaker, but he wanted to help fix it too. He was upset, the staff was upset... we all were. As you have seen in the letter from President Jon Dudas... FIRST is going to try to figure out what happened and make a solution. I fully believe that it's important to recognize each of the Division champion teams for their efforts on Einstein in some way... and then work on resolving the connection issues for next year. I had no bearing on what happened, but I am truly sorry it did. I'm sure FIRST will make sure this never happens again. |
![]() |
| Thread Tools | |
| Display Modes | Rate This Thread |
|
|