THE HORROR! THE HORROR!

[Team23pitboss]

Originally Posted by Kevin Sevcik

Black Jags are serial controllable and have current control modes plus built voltage and current feedback. So all you would need for a battery tester is a serial port (or USB-Serial adapter), a serial to Jaguar adapter, a Black Jaguar that most every FRC team has 1 of, and a power resistor you can get at Mouser or your local electronics surplus store. Battery on the input side, power resistor on the output side, tell the Jag to dump X amps into the resistor, then record and plot your feedback. Primary difficulty is making the program to control the Jaguar. Either I'd need to figure out how to generate the FRC heartbeat, or you'd have to flash the Jag with custom firmware.

Wow, that is an absolutely brilliant idea I think that our electronics guys would probably be interested in trying something like this out at some point. It also give me something to do with all the ancient computers with serial ports I have floating around my house.

[Ether]

Originally Posted by Team23pitboss

It also give me something to do with all the ancient computers with serial ports I have floating around my house.

You can turn those ancient computers with serial ports into test equipment, like for example a poor-man's logic analyzer to inspect the timing of digital signals like encoder pulses. Or DIO set by tasks to inspect scheduling timing and jitter.

[jspatz1]

Originally Posted by Team23pitboss

Out of curiosity did you ever figure out why the batteries behaved in such a strange way?

They were simply failing batteries/cells that managed to emerge at the worst possible time. You can be sure we are taking measures to ensure that does not happen again.

[epylko]

Originally Posted by Alpha Beta

Our spinning was due to a failed battery. We had 2 separate instances in elims where batteries showed 13 volts on the charger and dropped to 6 volts within seconds during autonomous.

Our team had a problem like that this year - except it wasn't a failed battery. The robot took a hit in an earlier match which made the DC connector on the AXIS camera touch the metal on the back of the camera (which is metal. Doh!) The back of the camera was touching the frame which caused some fun grounding issues. The driver station was showing the battery around 7V.

Fortunately, Rob the head robot inspector, saw this once before with an AXIS camera and was able to point this out to us.

[pfreivald]

Originally Posted by epylko

Fortunately, Rob the head robot inspector, saw this once before with an AXIS camera and was able to point this out to us.

Hah! 1551's pain was your gain! (Rob's first experience with the extreme wonkiness that can come from a frame-grounded Axis camera was at our expense.)

[Team23pitboss]

Originally Posted by Kims Robot

I highly doubt you can diagnose an FMS problem via a webcast. I know of only a handful of people that are even remotely qualified to do that - and they are called FTA's. I presume you are not one, thus I'm not sure how a robot dying on a field (Im guessing thats what you mean by "spin out") at their very first event can be assumed to be an FMS fault. Many teams have not finalized or fixed or even know if their code really works yet.


What evidence do you have of this? I hate to dredge it up, but the Einstein report showed that the majority of the issues were actually team faults. I highly suggest you go back and read it if you haven't already. It will give you some great insights into things that your team should not do if you want to have a successful fully running on field season.

Should there have been better diagnostics? sure. But its not as simple as saying "the FMS ended our season." Maybe in your case it was one of the rare instances it was true... but I don't see how we can/should panic over seeing a couple of dead robots on a webcast.

To fill in the blanks about what exactly happened to us last season, it was the third elimination round of the quarter finals, autonomous had just ended when suddenly the robot began to spin wildly in circles and continued to do so until the end of the round, eliminating us from the competition. The team was incredibly distraught and was disappointed to have lost due to something so completely out of our control.

My panicking was meant to be taken with a grain of salt and was extremely sarcastic, hence the memes and star trek GIF. However, my questions about the integrity of the FMS last season (and possibly this season) is quite serious. While I make no claims to be able to diagnose an FMS failure over the web I can say with a high degree of certainty that our failure at WPI last year was not based on a fault in our code. We functioned normally for the entirety of the competition and had not experienced any previous failures. The FTA's at the event looked over our watchdog logs and our robot code at length and were incredibly helpful in trying to deduce what had caused our problem. They agreed with us that it was not an error in our code and even tried to get us a rematch to no avail. We have tried on multiple occasions to replicate our spinning on the field and have been unable to do so.

I trust enough in the folks behind FIRST to believe that the FMS will be a non-issue but our experiences last year will never be too far from my mind.

[Kims Robot]

Originally Posted by Team23pitboss

We functioned normally for the entirety of the competition and had not experienced any previous failures.

This actually sounds EXACTLY like the situation that much of the Einstein report pointed out, as well as several other teams noted during the season. The most well known of these probably being 118. They functioned "completely normally" until the elims in CT. They also functioned "completely normally" until Einstein. Again, I reference the Einstein report. I highly suggest you go and read pages 12-13 [and really the rest of it, but specifically those pages if you don't want to learn from the whole thing].

And I appreciate that different people have different senses of humor, and that your graphics should have indicated your tone of sarcasm... but as the saying goes "TOO SOON...TOO SOON".

[arun4444]

We had problems also with all robots shutting down and having to play a rematch.

Not sure why frc still uses wifi, should move to independent channel RF.

[stingray27]

Originally Posted by Team23pitboss

I really hope I'm wrong but it looks to my untrained eyes like the FMS could possibly be as unstable as it was last season. I have already watched multiple robots begin to spin wildly mid-match while watching the Florida and Texas streams. If you don't already know Team 23's season was brought to an end by a round loss due to a possible FMS malfunction in the quarter-finals last season and the team has been especially wary of a repeat of this ever since. Do you think the FMS will be an issue as it was for some teams last year and if it is what can FIRST do about it?
http://s1.postimage.org/y2y5izgy5/Not_0a1d39_92289.gif

Team RUSH 27 ran into Smart Dashboard issues at Northern Lights that spammed our cRIO and ran our processing to over 100%. Blame was originally on 7Mbps but after further investigation, NI advisors suggested it may be an issue with the smart dashboard itself. This caused major issues that hindered our robot for 3 matches. Any other issues similar?

[coalhot]

Originally Posted by Team23pitboss

I really hope I'm wrong but it looks to my untrained eyes like the FMS could possibly be as unstable as it was last season. I have already watched multiple robots begin to spin wildly mid-match while watching the Florida and Texas streams. If you don't already know Team 23's season was brought to an end by a round loss due to a possible FMS malfunction in the quarter-finals last season and the team has been especially wary of a repeat of this ever since. Do you think the FMS will be an issue as it was for some teams last year and if it is what can FIRST do about it?

I'd like to ask that if you'd like to blame the field for the issues, you should post the code and pictures of your electronics/wiring so that we could look at it. You can't criticize the field without full disclosure of the code. I say this because I was talking to a few CSA's at a week two event, and more than half of the issues with FMS malfunctions were robot-based and not field-based. Many of us are eager to blame the field equipment on faults, but that's rarely the case. Usually it's something with the robot.

Also, what's supposed to happen when multiple robots trip the bandwidth limit?

[Nuttyman54]

Originally Posted by coalhot

I'd like to ask that if you'd like to blame the field for the issues, you should post the code and pictures of your electronics/wiring so that we could look at it. You can't criticize the field without full disclosure of the code. I say this because I was talking to a few CSA's at a week two event, and more than half of the issues with FMS malfunctions were robot-based and not field-based. Many of us are eager to blame the field equipment on faults, but that's rarely the case. Usually it's something with the robot.

"Something with the robot". I really, really hate that teminology. It is entirely too general, and has connotations that it is something in the team's code. Anything between the DS and robot is not the FMS, true, but that does not imply that it is something the teams have control over. Case in point is the C++ issue in SmartDashboard that they released a bug fix for in Team Update 2013-03-05. The bugs affected teams at Week 1 events, but were not part of the FMS. Yes, a lot of times it can be something in the team's code, but there can be "robot side" issues that are not the team's fault or responsibility to fix. It is very frustrating for teams to encounter these, and be told "It's something with your robot", but have no ability to diagnose or fix the problem. The term is general, and can apply to anything that is not due to the FMS, regardless of who has ownership of the buggy code.

Originally Posted by coalhot

Also, what's supposed to happen when multiple robots trip the bandwidth limit?

According to the FMS whitepaper, the FMS puts a priority on robot control and status packets, so any other packets are likely to be dropped. Trip times will also increase drastically above 6mb/s, and the team exceeding their bandwidth cap may experience lag.

[akoscielski3]

I was behind the FTA table at GTR east this weekend learning how to be a score keeper. From what I could tell any problems that happened was because of the teams error. there was one team in particular who was dieing in the middle of matches all the time. The FTA's went to their pit and discovered the problem was in the program, (i'm not a programmer so sorry if I'm wrong) there was a problem where the amount of time between the robot reading the code was too slow. However the Programmer Refused to change the code for some stupid reason and kept blaming it on the field.

In FLR we had a problem where we died in 2 of our matches. the first match that it happened the FTA's quickly hurried to our pits to try and figure out the problem with our robot, as they knew the problem was not in the FMS. I am very pleased with the FTA's this year, and I am sure they don't want to have a year like last year. We figured out that the problem was the cRIO wiring, the two wires powering the cRIO were so close together that when we got hit by another robot we would die.

The FMS system is working perfectly fine to what I can tell. For some reason it seems like some people are having issues with their code, where they get stuck in a "loop" and then their robot is confued, dies, or does the same thing it was just doing. Again thats the teams fault not the FMS

[coalhot]

Originally Posted by Nuttyman54

"Something with the robot". I really, really hate that teminology. It is entirely too general, and has connotations that it is something in the team's code. Anything between the DS and robot is not the FMS, true, but that does not imply that it is something the teams have control over. Case in point is the C++ issue in SmartDashboard that they released a bug fix for in Team Update 2013-03-05. The bugs affected teams at Week 1 events, but were not part of the FMS. Yes, a lot of times it can be something in the team's code, but there can be "robot side" issues that are not the team's fault or responsibility to fix. It is very frustrating for teams to encounter these, and be told "It's something with your robot", but have no ability to diagnose or fix the problem. The term is general, and can apply to anything that is not due to the FMS, regardless of who has ownership of the buggy code.

While you may hate that term, it's still a fact that many matches were replayed due to code (such as the camera settings being wrong). Streaming max resolution at 30Fps will tend to kill the connection to your robot, and because of odd circumstances, can even affect the rest of the field. And yes, the C++ issue wasn't any team's fault, but FIRST didn't know about it/couldn't release a patch till week two.

My point still stands: the FMS isn't the cause of most of the issues this year. From what I've seen, it has almost always is something in the code, or on the robot. Now, I'm not saying that it's always the fault of the robot. The Field breaks sometimes. However, the majority of issues so far from what I've seen (this year) have been code related. This is why I asked for the OP to post code and photos of the wiring, not only for us to be able to say "that's what caused the issue" (if it happened to be the robot or code), but also to look at it and say "that's something I want to avoid doing with my code/wiring/robot, so I don't have the same issues"

Originally Posted by Nuttyman54

According to the FMS whitepaper, the FMS puts a priority on robot control and status packets, so any other packets are likely to be dropped. Trip times will also increase drastically above 6mb/s, and the team exceeding their bandwidth cap may experience lag.

If this is the case, why does a field with say three robots that exceed the cap happen to kill the whole field? It should only cause the offending robot/s to cut in and out, right? Or am I missing something?

It's funny though, the camera seems to be the bane of all field issues so far this year (spotlighted by the FTA at NY regional having all the teams turn off their cameras for eliminations). What has changed between the past few years and 2013 that could cause issues like this, other than the bandwidth cap?

[Nuttyman54]

Originally Posted by coalhot

If this is the case, why does a field with say three robots that exceed the cap happen to kill the whole field? It should only cause a single robot to cut in and out, right? Or am I missing something?

It's funny though, the camera seems to be the bane of all field issues so far this year (spotlighted by the FTA at NY regional having all the teams turn off their cameras for eliminations). What has changed between the past few years and 2013 that could cause issues like this, other than the bandwidth cap?

You are quite correct that the vast majority of team issues are in fact problematic code on their robot. My point wasn't directed at you, but more at the general way that everyone seems to say "well, it's not the FMS so it must be your robot". The system is far more complex than that, and it's time we as a community start respecting that sometimes, just being not-FMS related does not indemnify FIRST from responsibility for the issue. Likewise, however, the issue IS very rarely FMS, and when it is the FTA can usually identify that.

I don't know why teams exceeding the cap are affecting the rest of the field. It's not my area of expertise, and the FMS whitepaper does give many details on how the bandwidth cap is implemented. It does not seem to be working correctly, or at least how any reasonable person would expect. The logical assumption is that all teams are allocated 7mb/s, and any team exceeding that will be throttled so as not to affect the bandwidth of the other teams on the field. Any usage over 6mb/s sees a sharp increase in trip time, and will result in possible control lag (presumably only for the team nearing their limit).

I would say that the camera is an issue primarily because of it's increased usefulness and the ability of teams to stream live feedback to their robots. In 2009, 2010 and 2011, this ability was not particularly useful and was note widely used. Last year, it was extremely useful for vision tracking or just for lining up shots. This year, I suspect even more teams have started putting robot-eye feedback, which seems to result in problems when several of these robots are on the field at once. The bandwidth cap may also be affecting things, since it limits each team to ensure that all teams will have an equal share, rather than dynamically reallocating bandwidth up to the maximum the system can handle to accommodate for a few robots using more than their even share.

[DominickC]

Originally Posted by coalhot

I'd like to ask that if you'd like to blame the field for the issues, you should post the code and pictures of your electronics/wiring so that we could look at it. You can't criticize the field without full disclosure of the code. I say this because I was talking to a few CSA's at a week two event, and more than half of the issues with FMS malfunctions were robot-based and not field-based. Many of us are eager to blame the field equipment on faults, but that's rarely the case. Usually it's something with the robot.

Unfortunately we will not be posting code or pictures of our wiring.

Later inspection from the FTA, CSA, as well as a National Instruments rep yielded no result. The code was ruled out as a non-issue.

EDIT - We later attempted to replicate the failure using FMS simulation software provided to us by the CSA. All attempts at replication failed.