FIRST please fix....
 

First please fix auton- telop - auton bug before worlds. If you can't fix it at least get the old fog horn to see it and stop the match. Thank You

Re: FIRST please fix....
 

We are in the process of gathering our driver station logs from SVR to hopefully help FIRST hunt down the bug. I recommend that you do the same.

The behavior we saw was that 1.5 seconds into autonomous mode, the field would transition into "teleop disabled" for 0.5 seconds, then resume auto mode for 8 seconds. This happened in two of our matches, and has apparently happened at other events as well (for example, we saw this issue in 118's logs from one of their events).

Re: FIRST please fix....
 

This is the same behavior we saw in MSC QF1-1 and it's first replay.

Re: FIRST please fix....
 

http://youtu.be/HH_CG-Jmpr8?t=1m

I think that might have happened in QF1-2 at Milwaukee. It looks like all the robots stop at once, and then we(1675; big yellow PVC forks pointing up) ram our alliance partner. Same code, stopped in match 1, not in 2. This behavior didn't make sense at all to me, or anyone on our team at the time, but this field issue might explain it. Perhaps it screwed up our Timer or something.

Re: FIRST please fix....
 

This happened to me not once or twice but every single time we played on the field but i also seem to have the bug within my practice mode of my DS. Ill be sending my data as well. I did some figuring with a dumb luck trick involving the delay command in java in order to prevent the bug from causing major issues. I can't exactly explain it very easiily because im not sure how or what I did to fix it. Ill post my logs and code in this thread to see if you all can benefit from it.

My guess is that any time the delay command is running it ignores anything its told from the fms.

Re: FIRST please fix....
 

Greg McKaskle from National Instruments was on site at MSC this weekend. He and the event FTAs were able to capture a log of this glitch event happening in QF1.1. Hopefully this will provide sufficient detail for FIRST to root cause and solve this issue.

Re: FIRST please fix....
 

From the other alliance's point of view, I could see this being a possibility as well. One of our alliance partners, 2039, drove a bit farther than they usually did, so they missed their shot. Our team, 537, also experienced problems as our robot never fired its ball during autonomous and drove a little bit farther than normal. Maybe the encoders on 2039 and 537 were reset part way through. I could see that possibly causing these problems, as 2039 drove farther and we never reached the encoder value that we fire our ball at.

Re: FIRST please fix....
 

Shouldn't the FTAs call a field fault and stop the match right after the glitch occurs? The second replay of QF1-1 at MSC should have been stopped much earlier.

Re: FIRST please fix....
 

Is this the same as auto being disabled after a second? I might have the logs for this but I don't know why autonomous would just be cut short 9 seconds. Additionally the match wasn't replayed(would not have changed outcome).

Here is a link to a match video where auto was cut short to 1 or 2 seconds. No robot shot a ball this match and I have seen working autonomous from our robot and others in previous matchs.

http://www.watchfirstnow.com/archives/90605223

Re: FIRST please fix....
 

If they catch it, yes. However, we don't know if the FMS is reporting this hiccup (I didn't see this at all at 10,000 Lakes or Lake Superior). If the FMS doesn't show it and no one reports it, the field staff won't know that anything happened.

The other issue is that if it is an FMS glitch that happens in every match, they really have no choice but to keep playing the matches with the glitch (with reasonable warning to the teams of course).

Re: FIRST please fix....
 

They replayed QF1-1 twice due to the issue. Any observer could have predicted the possibility of the glitch reoccurring and seen the effects of it if it did. Instead of stopping the match after the red alliance missed once more, the FTAs decided to let the match run it's course.

Re: FIRST please fix....
 

I think this explains what happened to us in QF-2-1 at Northeastern. Auto appeared to reset a couple of seconds after the match started. Our robot ended up driving into the wall (the encoders reset at the beginning of auto) and never shooting (since it never got to the correct number of encoder ticks). I think 1761 missed their auto too. No replay, but luckily for us we won the second two matches. As far as I know it never happened to us again.

Re: FIRST please fix....
 

Off topic: that blue human player went just a bit out of his box...

On topic: I guess the one advantage to the java template we used was that if auto was started and interrupted (either going to disabled or teleop) it would actually pick up where it left off when it was enabled again.

Re: FIRST please fix....
 

We asked that question to the FTAs. They told us if they stop the match then no match log files are created. So they kept the match going to collect additional data to help determine the root cause of the problem.

They told us before the third match that they would stop it early if it happened again since they weren't going to learn anything new by continuing with the match.

The second time the issue showed itself slightly longer into auton (maybe 6-7s in), it almost seemed to be some sort of interaction between our CheezyVision timing and the field....since both times it seemed to occur right when we selected the HOT goal. We were going to stop using it, but the FTAs wanted us to keep everything exactly the same.

We ran the same auto for about 8 qual matches and every single elim match....so I don't know why the outcome would have changed.

It'd love to know when the true root cause is determined and see the data that proves it.

Re: FIRST please fix....
 

Sorry to derail the thread but there are no rules that say he can't go outside of his box per G39. As long as he is in contact with the human player area he started in, I can't find any rule that he is breaking. Unless you're saying he lifts his left foot every once in a while and loses contact. You may be right about that.

G39
During the MATCH, TEAMS must remain in contact with the area of the FIELD (ALLIANCE STATION or HUMAN
PLAYER AREA) in which they started the MATCH. Exceptions will be granted for inadvertent, momentary, and
inconsequential infractions and in cases concerning safety.

Re: FIRST please fix....
 

G39 is "remain in contact". While he kept a foot in the area of the blue box, there were times when that foot was off the ground.

I interpret those times when a part of the body is in the area bounded by the box as "Exceptions will be granted for ... momentary". That allows for a player to jump up to catch a ball, while fully staying in the box.

Re: FIRST please fix....
 

Here are some DS logs from matches where this happened to us. Q92 and QF1-1 at SVR.

I've also attached a log from team 118, who saw the same bug at Alamo.

Re: FIRST please fix....
 

I would be very surprised if CheesyVision was responsible. There are only ~30 payload bytes (plus TCP overhead) being transmitted to the cRIO per second on one of the open ports listed in the manual. That is peanuts compared to the live video streams and Network Tables data that teams have used successfully for years. Besides which, we have evidence that this bug occurred prior to CheesyVision ever being used. I am confident that this problem is unrelated and will be fixed before Championships.

(But just in case, we are going to make our autonomous routines robust to the problem...)

Re: FIRST please fix....
 

I'm glad Greg was there to see it and record info on it. We were all questioning what would happen if they were unable to fix the bug - indefinite replays or just let the defective matches stand? Yikes!

I am ALSO very glad that Marjie Jenkins took the microphone and called out booing after the the second replay was announced. It doesn't matter which team in particular, but I hope the mentors on that particular team have a long talk about gracious professionalism with their members. I'm sure their students would have felt differently had they been on the other side of the field glitch.

I believe Greg grabbed our DS logs as well - we had actually shown him some earlier when he was looking at the field networking.

Re: FIRST please fix....
 

Could you go into detail about how you plan on doing that (because 254 apparently hasn't helped our auto enough yet this year:rolleyes:)?

Re: FIRST please fix....
 

I'm sure 254 can go into detail on their particulars but here are some general principles:

• Add a hasRun flag into autoInit to sure any macro initialization runs once (resetting gyros and encoders for example)
• Make sure any PID with an I component behaves appropriately if the robot is disabled briefly. This may be the case already or it may require making sure that the PID is disabled when the robot is disabled or resetting it when the robot is enabled to avoid integral wind-up.
• Implement the auto routine as a state machine (as opposed to a hardcoded sequence). If using Iterative Robot, you're probably OK already in this regard. If using Simple Robot you may not be. I'll have to go back and look at some of the details of the Command Based implementation to give proper advice there.
• If using the Simple Template in C++ or Java, make sure to save the above state data in global variables (not in the auto method itself). This allows the state to be maintained if you drop out of the auto method.
• Again if using the Simple Template, make sure you have implemented the state machine above as a loop which checks that the robot is enabled. This is important to make sure that you are not taking actions and moving through states which have no external feedback with the robot disabled (and therefore not actually able to take these actions).

Re: FIRST please fix....
 

Is this bug localized to one single robot or an entire alliance? We had an instance in Calgary where our autonomous shot one ball then the shooter failed to reload, (to shoot the second ball) the intake mechanism seems to cycle yet cannot due to the shooter arm blocking it, then rather than pausing, triggering the shooter and then driving forward for mobility- as soon as the intake appears to deploy and retract, it immediate skips to driving forward. We were unable to reproduce the behavior off the field in practice mode and never figured out what caused it. It only happened the one match. I wonder if this was this bug in action.

http://www.watchfirstnow.com/archives/91370417

Our typical autonomous follows:
*enable*
>open pneumatic latch releasing spring-loaded shooter arm to fire 1st ball
>reload shooter
>>activate loading arm to charge catapult
>>when latch limit switch triggered by catapult, close pneumatic latch
>>reverse loading arm to move it out of way of catapult until it hits limit
>deploy intake mechanism with intake wheels running
>retract intake mechanism and turn off wheels
>pause 1.5 seconds for 2nd ball to settle
>release pneumatic latch to shoot second ball
>drive forward 1.5 sec

In that match it seemed to follow: (Red=skipped steps)
*enable*
>open pneumatic latch releasing spring-loaded shooter arm to fire 1st ball
>reload shooter
>>activate loading arm to charge catapult
>>when latch limit switch triggered by catapult, close pneumatic latch
>>reverse loading arm to move it out of way of catapult until it hits limit
>deploy intake mechanism with intake wheels running
>retract intake mechanism and turn off wheels
>pause 1.5 seconds for 2nd ball to settle
>release pneumatic latch to shoot second ball
>drive forward 1.5 sec

When teleop enabled, the driver was able to click the fire button and the system reloaded automatically as usual. This means that the shooter's reload mechanism had not moved from its home position on the limit switch and had not even attempted a reload cycle after the first shot. It's almost as if we had forgotten to put the machine in 2-ball autonomous however, the intake mechanism cycling proves that it was indeed trying to go for a second ball.

Re: FIRST please fix....
 

At MSC, it was the entire red alliance in QF1-1 and QF1-1.1. They appeared to replace every connection to the FMS (all the green Cat5e cables) before eventual third and successful run of the match. I know they were on the horn with FIRST HQ to help troubleshoot the issue. Hopefully this is something that gets resolved before then. I can't believe a system was designed that doesn't allow them to kill a match early.

Re: FIRST please fix....
 

They can. See Adam's post above for why it wasn't the second time (the first time presumably the FTAs were not aware there was an issue)

Re: FIRST please fix....
 

Oh, i realize they "can", but i find it odd it doesn't write the logs when they call a field fault and kill the match. Then again, I'm not a programmer and have no idea what that entails::rtm::

Re: FIRST please fix....
 

The issue isn't an inability to kill a match early; it's the inability to find out what went wrong (by looking at the logs) if a match with an error is killed early.

Re: FIRST please fix....
 

I'm pretty sure no one individual team was responsible for the booing, but rather a collective disappointment from a large portion of the audience. It wasn't directed toward the teams or the staff, but rather HQ itself.

I fully understand that everyone involved with FIRST gets nothing in return for sacrificing their time to make this a possibility except for the satisfaction of helping a lot of students, and I am eternally grateful each of their services. This couldn't be done without them. I myself am a volunteer, both within my own teams and at the events. But come on guys, it's week 7 already! FRC is a product that we all buy into and expect to work, and this kind of problem this late in the season is simply unacceptable. Chances are it had nothing to do with any of the volunteer staff on site; they're just doing their job the best they can. Maybe it's just because I'm getting old enough to finally noticing these problems, or maybe this is actually new, but while all the foot soldiers plod along at each event doing their jobs excellently, quality from the top of the organization seems to have declined.

I'm not mad at the GDC for some of the questionable decisions they made this year regarding the unnecessarily large responsibility allocated to the refs, or some of the penalty values (or the refs for enforcing these penalties), or even the large push toward heavy defense or 1 game-piece games, but this kind of technical failure bothers me greatly.

Keep in mind I have no horse in this race (I was completely unaffected by any of these technical failures and my teams were eliminated from Championship contention weeks ago). I just feel bad for all the alliances including MSC QF 1-1 that have gotten screwed by the the field system we were given to use. I'm just frustrated that while all the other teams are doing their best and working their darnedest, considering the pedestal lights, the hot goal timing, match length discrepancy, and now this, the FMS we were given doesn't seem to be fully debugged before being released to us for use.

disagree with me or not, these are my individual feelings. I don't know any of the facts, so I may be entirely in left field. Thank you event-staff FMS guys for working your butts off to get it fixed. You guys are troopers. This kind of problem is WAY over my level of understanding.

Re: FIRST please fix....
 

We saw this exact same issue at North Star! There were two matches where our autonomous drove forward, put down our arm in order to shoot, and just stopped. We spent half a day trying to figure it out, and the only thing I could see were gaps in autonomous that looked exactly like that - with no apparent cause. It seemed to "fix itself" after those two matches and never happened again, so we pretty much forgot about it.

Re: FIRST please fix....
 

If you think you were affected by this issue, please run the following tool against your DS Logs: http://www.chiefdelphi.com/forums/sh...67#post1373367

It should be much faster then manually looking at log files and much better then guessing from video evidence.

Re: FIRST please fix....
 

What RufflesRidge posted is basically correct. Persist autonomous mode state across mode changes so you pick up from the same place you left off. In our case, we use a lot of timing from the start of the match that was thrown off by the false start of teleop.

Also, for those that asked: In our experience, the entire field experienced this issue simultaneously.

Re: FIRST please fix....
 

It would have affected everyone in the match at MSC but it only affected red alliance because their auton was based on time and not sensors. Well, something like that.

Re: FIRST please fix....
 

I have no clue if this would be something that the field caused but 3 times throughout the season(last time being at states), our auto runs twice for some reason. We tried to replicate it over and over again prior but could never do it...now I am curious if it was the field that caused it.

Re: FIRST please fix....
 

Something like that happened in QF1-2 at NYC. Our bot (far right in the link below) drives forward in Auto, stops, and then plows into the low goal without shooting. Prior (and subsequent) to that match, our Auto stopped the bot 1 ft shy of the low goal 100% of the time. Our alliance partners' auto was also interrupted (though its harder to see because they didn't have a ball).
http://youtu.be/DoGdrFZtdEI?t=2m2s

Sorry I don't have logs to post.

Re: FIRST please fix....
 

I was the scorekeeper at MSC and was an FTA for 3 FiM districts this year. I don't have the answers but I can clear up/confirm some things.

As some have mentioned, yes, you have to finish the match and commit the scores before the match logs are saved. That's why we had to complete the second match. FIRST HQ wanted the logs from both matches.

As Adam stated, we were going to stop the third match if we saw the same behavior.

Right in the field monitor logs it clearly showed that FMS transitioned the robots from Auto to Teleop back to Auto. It affected all 6 robots on the field. As mentioned in this thread, the actual outcome of that incorrect transition is different for each team based on their code implementation.

This is one of the purest definitions of a 'field fault' and is a replay no matter what teams are on the field, what match it is, or whether or not the match is played for 10 seconds or the full 2:30.

The 'signature' of the fault is known and when it occurs the data confirming it can be found in the field logs and the driver's station logs. I can't speak to whether or not FIRST can fix this before Championship but I'm sure they will try.

Re: FIRST please fix....
 

It's a pretty fair bet it happened there too - see the blue bot on the opposite side of the field that appeared to drive forward much further than it should have, then fired the ball into the wall?

Re: FIRST please fix....
 

Our software protects against this kind of field fault so I guess it's possible this happened to us but we wouldn't know. We've been doing it this way for years for various reasons.

Here's how we do it (in case anyone is interested):

1) All of the sensor zero-point calculation and resetting is done in disabled.vi

2) We keep a global variable called "VisitedEnabled" which gets initialized to FALSE. As soon as the robot becomes enabled, this variable is set to TRUE and can never become FALSE again without a cRIO reboot (or a debug-mode override).

3) In disabled.vi (or your disabled mode code) sensors are rezero'd/cal'd ONLY if VisitedEnabled is FALSE.

Re: FIRST please fix....
 

We always reset everything when we exit disabled mode (disabled bit goes from low->high). This is done as a safety features, so that the robot will never do anything when entering enabled mode. Autonomous is also reset when this happens. This is also a development feature, as we frequently run auto many times during calibration and we don't want to have to reset anything in the software realm to run again. If we are ever in a situation where the field disables us and robs us of a second of our 10 second autonomous mode, we will contest it as a field fault to the highest level required and demand a replay.

FMS stands for Field Management System. Its job is to Manage the Field, its FIRST job is to make sure the match runs properly (secondary tasks include saving scores, audience displays, ranking calculations, ...). This includes correctly forwarding communications between all teams playing (which it failed at the 2012 Championships), and correctly time the match (which it has failed at every event this year). There is absolutely no technical reason that the FMS should be more than 20ms off on any time event that happens during the match - The enabling transitions of the robots, the actuating of the hot goals, the lighting of the pedestal, etc. and yet the FMS still manages to screw it all up. We can get timing accuracy of 10ms on our robot controls, and we could achieve 5ms if we wanted to. I have systems at work and in FSAE that go down to 1us or less, on processor hardware that has been in production for around 15 years. Why can't the FMS maintain time to half a second?

FIRST is telling the teams (and the poor refs) that they are held accountable to every plane in space and many totally subjective calls, yet they can't maintain the match or hot goal timing to any reasonable accuracy by week 7. This is absolutely unacceptable. I am happy that the decision was made at MSC to replay the match as many times as necessary.

Re: FIRST please fix....
 

With the advantage of hindsight, it seems to me that they could have disabled all 6 robots after the faulty autonomous and let the match timer run out, this could have saved a lot of people from the emotional roller-coaster, and reduced the fatigue for the teams on the field.

Having to keep strategies, robots, and people fresh after a grueling 5 matches in quarter finals doesn't sound fun to me.

Re: FIRST please fix....
 

When testing autonomous modes back to back, do you always reboot the robot or use the debug override?

Re: FIRST please fix....
 

We use the debug overrides to reset everything back to power-up states. We do most of our autonomous testing while running through the play button so we can tweak PID gains while we run, and also so we can see what is going on internally in case we need to debug something.

The biggest reason we do it the way we do is we often use sensors in both autonomous and teleop and the robot is often moving at the end of autonomous. We don't want to do sensor re-cals in that brief period between modes just in case the robot is moving.

Note: we've had problems in the past with the WPI encoder reset and gyro zero/reset so we have our own encoder reset and gyro zero software that runs in disabled.vi. That's why it's important for us not to run that code between auton and teleop: the gyro re-zero code could screw-up our gyro zero-point if that ran while the robot was moving.

Re: FIRST please fix....
 

So as I was FTAAing/CSAing at Oklahoma regional I saw this same exact problem creep its head out in Final 1 though 3 with team 3528. Checked every safety config, every log though out all the matches. The interesting part was that auto either stopped then ran Teleop flawlessly or kept it in a disabled state.

The Question really for me is after knowing what teams like 33,254,1718 all do for optimization to the most. Why is it all at the same packet loss and CPU spike to entirely skip it and what in the framework provided by FIRST would allow the FMS to respond the way it does? This is all languages across the board.

My thought is that it has something to do with the Autonomous disabled transition to the Teleop enabled. I am very curious if it was from the software update that FIRST released after week 2?

Re: FIRST please fix....
 

Alamo was a week 1 event, so the log file from 118 above proves that this has been happening all season.

Re: FIRST please fix....
 

Ah thanks for that clear up.

Re: FIRST please fix....
 

I agree. With our robot, our code is written so that after the first autonomous mode runs, we must launch our debugging/tuning application and request another auto mode. Why should we have to remove safety features so that our robot works? The Fms seems like a horrible hacked together piece of software that isn't accurate or maintainable. I'm also curious as to why they decided to make the software delete logs from stopped matches. When a match is stopped, it is because of a field fault, so wouldn't it make sense to sAve logs from matches where the field didn't work? It would also be really cool if first was at least a tiny bit transparent about some of these issues. They could at least formally acknowledge that hot goal timing, pedestal timing, and match timing are all still flawed instead of just hopng that if they ignore their problems that they will go away.

Re: FIRST please fix....
 

This auto-stop-start might explain problems we had in OKC. But it did not happen at Dallas. We are looking at logs and will forward them to Joe Ross.

In our case the symptoms were a killer because of a custom script-driven auto sequence algorithm. We finally put the auto behaviour in its own task that would continue execution and pause/restart in a more robust manner.