THE HORROR! THE HORROR!

[Nuttyman54]

Originally Posted by coalhot

I'd like to ask that if you'd like to blame the field for the issues, you should post the code and pictures of your electronics/wiring so that we could look at it. You can't criticize the field without full disclosure of the code. I say this because I was talking to a few CSA's at a week two event, and more than half of the issues with FMS malfunctions were robot-based and not field-based. Many of us are eager to blame the field equipment on faults, but that's rarely the case. Usually it's something with the robot.

"Something with the robot". I really, really hate that teminology. It is entirely too general, and has connotations that it is something in the team's code. Anything between the DS and robot is not the FMS, true, but that does not imply that it is something the teams have control over. Case in point is the C++ issue in SmartDashboard that they released a bug fix for in Team Update 2013-03-05. The bugs affected teams at Week 1 events, but were not part of the FMS. Yes, a lot of times it can be something in the team's code, but there can be "robot side" issues that are not the team's fault or responsibility to fix. It is very frustrating for teams to encounter these, and be told "It's something with your robot", but have no ability to diagnose or fix the problem. The term is general, and can apply to anything that is not due to the FMS, regardless of who has ownership of the buggy code.

Originally Posted by coalhot

Also, what's supposed to happen when multiple robots trip the bandwidth limit?

According to the FMS whitepaper, the FMS puts a priority on robot control and status packets, so any other packets are likely to be dropped. Trip times will also increase drastically above 6mb/s, and the team exceeding their bandwidth cap may experience lag.

[akoscielski3]

I was behind the FTA table at GTR east this weekend learning how to be a score keeper. From what I could tell any problems that happened was because of the teams error. there was one team in particular who was dieing in the middle of matches all the time. The FTA's went to their pit and discovered the problem was in the program, (i'm not a programmer so sorry if I'm wrong) there was a problem where the amount of time between the robot reading the code was too slow. However the Programmer Refused to change the code for some stupid reason and kept blaming it on the field.

In FLR we had a problem where we died in 2 of our matches. the first match that it happened the FTA's quickly hurried to our pits to try and figure out the problem with our robot, as they knew the problem was not in the FMS. I am very pleased with the FTA's this year, and I am sure they don't want to have a year like last year. We figured out that the problem was the cRIO wiring, the two wires powering the cRIO were so close together that when we got hit by another robot we would die.

The FMS system is working perfectly fine to what I can tell. For some reason it seems like some people are having issues with their code, where they get stuck in a "loop" and then their robot is confued, dies, or does the same thing it was just doing. Again thats the teams fault not the FMS

[coalhot]

Originally Posted by Nuttyman54

"Something with the robot". I really, really hate that teminology. It is entirely too general, and has connotations that it is something in the team's code. Anything between the DS and robot is not the FMS, true, but that does not imply that it is something the teams have control over. Case in point is the C++ issue in SmartDashboard that they released a bug fix for in Team Update 2013-03-05. The bugs affected teams at Week 1 events, but were not part of the FMS. Yes, a lot of times it can be something in the team's code, but there can be "robot side" issues that are not the team's fault or responsibility to fix. It is very frustrating for teams to encounter these, and be told "It's something with your robot", but have no ability to diagnose or fix the problem. The term is general, and can apply to anything that is not due to the FMS, regardless of who has ownership of the buggy code.

While you may hate that term, it's still a fact that many matches were replayed due to code (such as the camera settings being wrong). Streaming max resolution at 30Fps will tend to kill the connection to your robot, and because of odd circumstances, can even affect the rest of the field. And yes, the C++ issue wasn't any team's fault, but FIRST didn't know about it/couldn't release a patch till week two.

My point still stands: the FMS isn't the cause of most of the issues this year. From what I've seen, it has almost always is something in the code, or on the robot. Now, I'm not saying that it's always the fault of the robot. The Field breaks sometimes. However, the majority of issues so far from what I've seen (this year) have been code related. This is why I asked for the OP to post code and photos of the wiring, not only for us to be able to say "that's what caused the issue" (if it happened to be the robot or code), but also to look at it and say "that's something I want to avoid doing with my code/wiring/robot, so I don't have the same issues"

Originally Posted by Nuttyman54

According to the FMS whitepaper, the FMS puts a priority on robot control and status packets, so any other packets are likely to be dropped. Trip times will also increase drastically above 6mb/s, and the team exceeding their bandwidth cap may experience lag.

If this is the case, why does a field with say three robots that exceed the cap happen to kill the whole field? It should only cause the offending robot/s to cut in and out, right? Or am I missing something?

It's funny though, the camera seems to be the bane of all field issues so far this year (spotlighted by the FTA at NY regional having all the teams turn off their cameras for eliminations). What has changed between the past few years and 2013 that could cause issues like this, other than the bandwidth cap?

[Nuttyman54]

Originally Posted by coalhot

If this is the case, why does a field with say three robots that exceed the cap happen to kill the whole field? It should only cause a single robot to cut in and out, right? Or am I missing something?

It's funny though, the camera seems to be the bane of all field issues so far this year (spotlighted by the FTA at NY regional having all the teams turn off their cameras for eliminations). What has changed between the past few years and 2013 that could cause issues like this, other than the bandwidth cap?

You are quite correct that the vast majority of team issues are in fact problematic code on their robot. My point wasn't directed at you, but more at the general way that everyone seems to say "well, it's not the FMS so it must be your robot". The system is far more complex than that, and it's time we as a community start respecting that sometimes, just being not-FMS related does not indemnify FIRST from responsibility for the issue. Likewise, however, the issue IS very rarely FMS, and when it is the FTA can usually identify that.

I don't know why teams exceeding the cap are affecting the rest of the field. It's not my area of expertise, and the FMS whitepaper does give many details on how the bandwidth cap is implemented. It does not seem to be working correctly, or at least how any reasonable person would expect. The logical assumption is that all teams are allocated 7mb/s, and any team exceeding that will be throttled so as not to affect the bandwidth of the other teams on the field. Any usage over 6mb/s sees a sharp increase in trip time, and will result in possible control lag (presumably only for the team nearing their limit).

I would say that the camera is an issue primarily because of it's increased usefulness and the ability of teams to stream live feedback to their robots. In 2009, 2010 and 2011, this ability was not particularly useful and was note widely used. Last year, it was extremely useful for vision tracking or just for lining up shots. This year, I suspect even more teams have started putting robot-eye feedback, which seems to result in problems when several of these robots are on the field at once. The bandwidth cap may also be affecting things, since it limits each team to ensure that all teams will have an equal share, rather than dynamically reallocating bandwidth up to the maximum the system can handle to accommodate for a few robots using more than their even share.

[DominickC]

Originally Posted by coalhot

I'd like to ask that if you'd like to blame the field for the issues, you should post the code and pictures of your electronics/wiring so that we could look at it. You can't criticize the field without full disclosure of the code. I say this because I was talking to a few CSA's at a week two event, and more than half of the issues with FMS malfunctions were robot-based and not field-based. Many of us are eager to blame the field equipment on faults, but that's rarely the case. Usually it's something with the robot.

Unfortunately we will not be posting code or pictures of our wiring.

Later inspection from the FTA, CSA, as well as a National Instruments rep yielded no result. The code was ruled out as a non-issue.

EDIT - We later attempted to replicate the failure using FMS simulation software provided to us by the CSA. All attempts at replication failed.

[Gary Dillard]

Our robot just did the circle dance for no reason. We tested code before the match and no problems. It started when autonomous started, then continued in teleop so we killed it. After long discussion with FTA about this thread, we powered down and went to the pits. When we started up in the pits it did the same thing. We checked grounds and vacuumed out the robot, no change. We redeployed the same code and then it worked fine. We had Rick Foley check the code and it all looks fine. Hmmmmmm....

[Brandon Holley]

The "system" (FMS, robots, programming languages, vision processing, CAN, etc) is very complicated, and needs to be respected as Evan stated. We have a natural tendency to draw conclusions by process of elimination, ie: if its not this thing it MUST be this other thing. With complex systems, this technique tends to not work as desired and in our case leaves people feeling helpless. I totally agree with Evan in that just saying to a team "its your robot" is an absolute disservice. It very well might be their robot, in fact 95% of the time it probably is their robot.

However, we as a community and FIRST as an organization have a duty to help get teams working correctly. We (FIRST and its community) have a duty to educate people wherever we can and to help bring people up to speed with this system and how to troubleshoot it. Do you think every team knows that their 30fps camera is bogging down the entire field? In my experience the answer is an overwhelming NO. This is the system FIRST has chosen for all teams, so its on their shoulders to help make this education possible (and have done a decent job as time has progressed with it.)


All that being said, cries of "FMS is messing up our robot!!" may be shortsighted, but are not always unfounded. This all stems from the fallout of Einstein and the investigation that followed. We must stress patience on BOTH sides (volunteers and teams) and understand we all have one common goal: to get every robot running through FMS on the field flawlessly. How the robot performs is up to the individual team obviously, but we all need to work to get teams operational at events.

-Brando

[Radical Pi]

In my experience, often when people blame the FMS for their problems, they really should be looking at the overall setup of the robot on the field. By simply running it in the pits and saying "it works fine, must be a problem with the field", a whole host of diagnosable issues are being ignored. For instance, by default any robot using RobotBuilder will have their main loop directly tied to the packets coming from the Driver Station. When the robot is operating over wireless, there's many more dropped packets, and each dropped packet is one loop of the robot code that doesn't run. An overly strict watchdog could easily cause problems there.

Case in point, last year at champs we had an issue where auton would work fine in the pits, but on field it would just sit there. It had worked at our previous regional, and under FMS lite we couldn't replicate the problem. Nobody could figure out what was up, and as much as I would have loved to blame the Newton field, the issue turned out to be a seemingly unrelated change to the code which canceled the autonomous command if the Driver Station wasn't connected when the cRIO booted. Since the DS was always tethered in the pits, but on the field it had to wait for the radio to finish connecting, the problem only showed in the pits if we knew what to look for.

[Greg McKaskle]

Our robot just did the circle dance ...

I assisted a team in Lubbock after a similar issue. It happened at the end of Th and we didn't find it. It happened again on Fri and we saw that most of the motor controller weren't receiving a signal when code was enabled. We discovered that the ribbon cable was no longer zip tied down. It was lifted on one side, so some connections were present, some weren't.

I've also seen this when a limit jumper "fell off" of a jaguar. Without the jumper, the motor will not run in one direction.

If you haven't done it already, check the basics, verify both the code, the electrical, and the mechanical. Could a bad sensor value cause this? Gyro's are finicky if not calibrated when the robot is stationary.

I'll be happy to help interpret the log file if you want to PM me.

Greg McKaskle

[mtaman02]

Originally Posted by Kevin Sevcik

I'm told many problems at Lone Star were caused by multiple teams trying to run cameras at 640x480 at 30fps. Or in one case, TWO cameras at that rate. Teams have been instructed to throttle down to 320x240 @ 15fps and things have calmed down a good bit.

^^^
Had the same issues with the NYC Regional with the cameras. Once we tamed the settings the field ran the way it did the previous 2.5 days before. Also had to look for a couple of other things that tied up the comm bands.


I don't believe there will EVER be a comm problem free season. Thats the nature of technology. The field and FMS always gets heavily looked at both locally and through a remote connection whenever a field allegedly has issues. Nothing is fool proof - all that can happen is that it improves a little with each passing day.