Discrepancy at Chesapeake, Israel, Waterloo?

[Alan Anderson]

Originally Posted by Dave Flowerday

We lost almost an hour at BMR due to the failure of FMS and the field.

It's more than a little unfair to call a complete loss of externally provided power to the entire pit and field a "failure of FMS".

For those of you complaining about the time it takes for the 2009 FRC control system to "link up with the field", my observation is that it's really a very quick process, on the order of five or ten seconds. What takes time is the cRIO booting up and starting to run the user code. The WGA is fully active and in the loop long before the cRIO even starts looking for the DS. I doubt that the FMS can tell that a team has managed to leave the network cable disconnected between the cRIO and the WGA until 30-45 seconds after the robot has been powered on. It isn't the FMS you need to be focusing your attention on; it's the cRIO. (Maybe the boot time can be reduced in the future by implementing something like a "hibernate" mode, or even a precomputed RAM image deployed as part of the program load process.)

[Dmentor]

This thread seems to have veered onto a tangent. I can't comment on what occurred at Chesapeake but I wanted to confirm Kyle's statement regarding NASA/VCU.

Originally Posted by nukemknight

At VCU, we did everything possible on field to get every team's communication linked. If their radio was unplugged, the field crew plugged it in. If the robot's power needed to by cycled, we did. We made every effort to have every team play in every match (Just ask the 3rd seeded alliance of 612, 2068, and 1908). If there was an issue other than the radio, we bypassed the robot for that match and directed the team first to inspection to have their game adapter checked for proper configuration, and then to the NI representative. After reprogramming, teams that dropped communication randomly during the match had further problems (ask 2108, 1222).

Even after all the problems with unplugged radios and taking the time to fix them, we still finished Friday ahead of schedule!

The field crew and referees were fantastic at NASA/VCU (well actually pretty much everything was awesome at NASA/VCU except getting beat in the semifinals again). They were helping to fix problems all regional long. When 1908's gaming adapter had to be replaced before the quarterfinals, Kyle patiently confirmed the configuration and assured us that he would do everything possible on the field to ensure that they were working properly. And he did, thanks Kyle! While 1908's robot was immobile for nearly all of QF 4-1, I'm convinced it had nothing to do with the gaming adapter.

In practice at DC, we had discovered that we always had to start the DS first, wait until it finished booting, and only then power on the robot. If we didn't follow the procedure, we would often experience odd robot behavior (sometimes we would be immobile, sometimes only our autonomous would work, sometimes our DS digital IO wouldn't work even though everything else worked fine, etc.) At DC, we could recreate this condition while tethered in the pits so we developed a startup procedure. At NASA/VCU, we continued to follow this procedure and it worked great until Q38 where we were once again immobile during the whole match. Of course everything checked out just find in the pit, but in the debrief with the drive team, the only deviation from the startup procedure that we used in the pits and what was used on the field was that when the DS was powered on for Q38 it was not in a disabled state (??). We tried but couldn't reproduce this failure in the pits, but we subsequently modified our startup procedure to include ensuring that the DS was in a disabled state before powering on the robot. This seemed to eliminate any further odd robot behavior. After QF 4-1, we also made sure 1908 followed our startup procedure and they didn't experience any further odd behavior but the issue could just have been a lose wire.

Thanks so much to the NASA/VCU crew for ensuring that all the teams had the best possible opportunity to shine!

[Dave Flowerday]

Originally Posted by Alan Anderson

It's more than a little unfair to call a complete loss of externally provided power to the entire pit and field a "failure of FMS".

I'm not talking about the 20 minutes of power outage - I can't believe anyone would think I'd blame that on anything other than the burned up breaker that caused it. I'm talking about the 45+ minutes afterwards that the field could not be restarted and wouldn't work, despite the obvious multiple restarts of the whole system by the field crew (who did an excellent job dealing with the situation).

[Lil' Lavery]

This thread was somewhat painful to read. Teams are pointing at FMS/Volunteers, Volunteers are pointing at Teams/FMS, FMS is pointing at teams, etc.

I know some posters have been more forthright and accepted responsibility, but in general people are making much more sweeping generalizations than they should (including my intro).

Like most difficult problems, this one is multi-headed and problems exist in each area.

Teams:
Ultimately the ball is in your court to make sure your robot is ready. I know about and I feel the pain of having a knew control system, long labview downloads, and having another series of items to have to worry about each match. And I wish every event had the time and crew that would allow for the event to be streamlined and fair. But the simplest way to make that happen is to take care of stuff on your end.
Write a pre-flight checklist. Run through it before each and every match. Find ways that work to make sure your robot can sync with the field quickly and flawlessly. Help out your alliance partners (and preferably each and every team in FIRST) to do so as well.

Field Reset/FTA/Volunteers:
Your job is not only to help the event run in a timely manor, but to also help the teams get the most out of the event (though that applies both ways, nobody like events that run 4 hours over). I know this is hyperbole, but you could run 80 matches in 3 hours if you allowed 0 minutes for teams to get on the field in between matches. Yet nobody would be happy with that. Running on time is great, but so is having teams running.
Try to help teams ensure they're ready to go by the time they're setting their robot on the field. Queuers and reset volunteers can ask the teams who are in queue if their radios, DS, and batteries are plugged in. You can post signs by the entrance to the field to help remind teams of the same thing.

FMS/FIRST/Other affiliated Parties:
Yes this is a new system. Yes we all expect it to have its kinks. Yes, it can run well when everything goes right and the teams do everything right.
But this is the real world. Not everything will go right, and the system needs to be designed to incorporate that.
We need ways to help fix and diagnose connection issues and controls issues (both from the field and robot perspective) quickly and efficiently. Whatever of these can be implemented DURING the season (once properly tested), should be. We shouldn't wait until next year.
We need to find ways for the robots and field to connect faster/cRio boot faster.
We need to find ways for the field to be reset faster so the teams can start booting and connecting their robots sooner.
We need to make this whole process more team friendly, and find better ways for teams to fix whatever issues they can BEFORE they get on the field.


Pointing fingers isn't going to solve anything, and neither is getting your feelings hurt. Determining solutions and being pro-active is required, but so is determining what isn't working currently. No party should be happy with the current situation, or any situation like it. Every party involved should be working to make the process better, pick up their end of the bargain, and work WITH the other parties instead of against them.

[petek]

Originally Posted by Dmentor

(snip)...the only deviation from the startup procedure that we used in the pits and what was used on the field was that when the DS was powered on for Q38 it was not in a disabled state (??).

At the risk of driving this thread even further off track, I saw this condition once in Pittsburgh and it turned out to be a flaky DS Ethernet connector.

[ColleenShaver]

I do think the efforts of the NASA/VCU field crew should be applauded! It's great that they were able to support so many teams and still run a timely event.

I understand it is difficult, and where to draw the line between team responsibility and teaching a lesson is blurry at best. But all being said, a little leniency for the teams who I think have been incredibly patient adapting to this entire process is in order.

I wasn't on the field the entire time in Chesapeake, but I was there on the sidelines the 3 different qualification matches where one of our partners was disabled by the FTA.

In two matches, everything was hooked up correctly as far as everyone could tell. The robots could not communicate so they were disabled. After some follow-through by our programming mentor & them not working the next match, it was determined one team had a radio problem. Another was never solved.

But red or blue, to allow plug in or not, there are a couple things that did disturb me:
- In our other match, the team had not plugged in their radio after being tethered in the pit. It was discovered once the field was set and all other teams linked and ready. The team was called back out on the field, shown the problem, allowed to plug it in, and then the scoring table was told to disable them. It certainly wasn't a time saving measure! Why not let the kids play? Why embarrass them like that? I'm sure they felt bad enough. This was done repeatedly when these problems happened.

- Teams have no way to troubleshoot problems, particularly with wireless. They are only allowed to tether in the pits and then they step on the field and are told "it's your robot".... Well their only defense is "it worked in our lab" because it's their sole point of reference. I totally agree with Adam's point of getting some more diagnostic tools in so field staff can say "it's an issue with your XXX" and teams have a chance of fixing it! How can you make a checklist when no one knows what you're looking for, and much of it can't be tested?

In general, teams were told to turn on the robot and leave the field immediately (some at the threat of disablement).... no 'gates down'. While I can certainly appreciate the consistency with which the field was run, I don't think it served the interest of teams or time-saving.

I personally am willing to give FIRST the chance to work out the bugs of all this new stuff, but I do think those representing FIRST (paid, volunteer, etc) need to show teams the same bits of compassion. I congratulate the events that have found the way to balance all perspectives on the issues!

[pitzoid]

Originally Posted by Kingofl337

Log the following times per team
1. Field E-Stop Status Per Robot
2. Driver Station Connection -> Ping is fine
3. Wifi Bridge Connection -> Again Ping
4. System Watchdog
5. User Watchdog
6. Log The console data being sent from the cRio for each team.

Well I think everyone needs a reality check on this one.....

I think the Lil Lavery comments are typical of what happens in these threads, everyone starts making assumptions about something that is contributed and before you know it it turns into the "finger pointing" fest. And for the most part thats why people who are in the "know" on things won't comment because nothing positive will come of it.

I didn't say it was the teams fault, I didn't say it was the volunteers' fault. I said "I wasn't there" so I don't know what happened, most of this stuff is at the discretion of the volunteers at the event and we can only hope they did the right things based on whatever circumstance they were dealing with.

As for what FMS has for trouble shooting, well, its about 2X what the IFI systems had (I know, I paid alot of attention to it the 2008 season). We can do this based off of how versatile ethernet is compared to other communication systems. Everything listed above and more is monitored, logged, and warnings issued. If its not being used, it because the operators are missing something. Fact is, FMS won't even let you start a match unless all 6 teams are linked with bots (or bypassed). QOS is monitored to make sure that the link between operator and robot is less than a 25ms loop (there and back), averages about 23ms. Thats what the Green Blinking light is you see on the score keepers table is, indication to the MCs and Announcers that the teams are technically ready to play the match. If someone was bypassed, it was a call made by the field personnel based on whatever circumstance they were dealing with. Fact is, I've had alot of scorekeepers comment to me that they are amazed by the way FMS is helping to run events this year....

I think sometimes people forget that we are dealing with very complicated devices that are essentially all prototypes, many built by people who are just getting their feet wet with technology. These aren't consumer toasters you buy at the local Target and even those don't work all the time. So sometimes they work, sometimes they don't, if the volunteer field personnel spent time fixing every bot that was not working on the field when it comes out, the tournaments would not finish in the alotted time. So its all sorta a wrong if you do, wrong if you don't situation, i.e. a no win. Its just the reality of the situation.

And just so you know, I'm very much a proponent of all bots must run on the field during the tournament if we can make it happen, soo much so that when the PHX event got down to the last bot that hadn't run during a match yet (the team had alot of issues they were trying to overcome), we figured that the teams radio wasn't working correctly, we didn't have a replacement they could use, couldn't find any close to the venue (guess teams had already bought out Best Buy), so I drove 45 minutes Friday night (each way) to find a store that had a couple WGAs, paid for it myself came back and gave it to the team Sat morning so they could run their last couple matches, let them have it.

So I guess what I'm ptting across here is please don't make assumption about what I put out or FMS capabilities, if you have questions, I'll answer you. PM me if you want, I've tried to answer all the messages that I've got this year. I want nothing more than every FIRST team to have an awesome time. And for the most part we've acheived that this year. People forget how long it took to get acclimated to the new IFI system after the introduction in 2004 (it took years) we are doing our best to make sure we work out the kinks as soon as possible.

THX

[pitzoid]

Originally Posted by ColleenShaver

- Teams have no way to troubleshoot problems, particularly with wireless. They are only allowed to tether in the pits and then they step on the field and are told "it's your robot".... Well their only defense is "it worked in our lab" because it's their sole point of reference. I totally agree with Adam's point of getting some more diagnostic tools in so field staff can say "it's an issue with your XXX" and teams have a chance of fixing it! How can you make a checklist when no one knows what you're looking for, and much of it can't be tested?

Hey Collen, I'm glad you bring this up. I know this is an issue and we've hit it around alot. 4FX has a solution for it also. We have a new team sign we designed that has a screen on the back of it that can do custom messages for each team, now only to find the budget to get them deployed for next year

[bandducky511]

Originally Posted by Dmentor

This thread seems to have veered onto a tangent. I can't comment on what occurred at Chesapeake but I wanted to confirm Kyle's statement regarding NASA/VCU.


At NASA/VCU, we continued to follow this procedure and it worked great until Q38 where we were once again immobile during the whole match. Of course everything checked out just find in the pit, but in the debrief with the drive team, the only deviation from the startup procedure that we used in the pits and what was used on the field was that when the DS was powered on for Q38 it was not in a disabled state (??).

Thanks so much to the NASA/VCU crew for ensuring that all the teams had the best possible opportunity to shine!

Good job on making it to semi-finals! I remember going against your bot in Q38. It started during autonomous at the outpost and did move; however, it stopped moving after our alliance partner collided with your bot. There was a huge collision with 3-4 robots around that outpost that match (if I remember correctly). The immobile part may have been caused by the collisions :S
Also, the DS does not need to be enabled or disabled. However, before you turn the robot on make sure it is disabled or enabled and on tele-op. The field judges are supposed to automatically turn each driver station on disable and autonomous at the beginning of each match (from what I have seen)

[MrForbes]

Originally Posted by ColleenShaver

How can you make a checklist when no one knows what you're looking for, and much of it can't be tested?

I made up this checklist during Thursday of our first regional, I couldn't get anyone to use it, but here it is anyways

Place Robot
Hitch Trailer Securely
Battery Strap buckled
Plug in Battery
Check battery cables—bumper perimeter
Align Turret
Load Balls
Field says 1726 over our Driver Station
Power ON
Check Router for power/comm.
Electronics Cover secure

Comments?

[pitzoid]

Originally Posted by Alan Anderson

For those of you complaining about the time it takes for the 2009 FRC control system to "link up with the field", my observation is that it's really a very quick process, on the order of five or ten seconds. What takes time is the cRIO booting up and starting to run the user code.

WGA is ~15 seconds to boot
DS is 10 to 20 seconds depending on how fast it identifies its USB devices
cRio can boot in as fast as 10 seconds (what I've seen in the shop, mine don't have a bunch of user code in them though)

I've seen WGAs link with the field AP in <2 seconds, and then I've seen them take as long as 20 seconds in the field, and I've seen some that don't link at all It all comes back to how much RF interferrence the WGA is getting from the robot. I.e. where the radio is placed in relation to metal bulkheads, motor controllers, relays and a multitude of other RF noisey things on the bots. this has been a common theme in many of my posts. I saw someone used my "it worked in our lab" comment, and the fact is, maybe it did work in your lab, but was marginal "in your lab", get out to the competition field with other RF competing devices and maybe it won't work..... That was the point. If you have one point of referrence and you know someone else that literally had hundreds, I'd go with that advice.

RF is voodoo, many things can affect it, treat with as much respect as you can.....

[Kingofl337]

Pitzoid, if a log of each match with of all of this information is available the field staff doesn't know how to get at it, because every time a robot stops running they usually have no clue idea why. If you have put in place the tools to monitor the DS, Bridge and the cRio individually that capability is not being utilized.

I have never herd someone on the field say "The log indicates the bridge stopped responding 1min 35sec into the match, but the driver station was still responding." or "The robot stopped communicating with the field at 1min 36 seconds the bridge and the DS were still responsive, at that time the console log shows the cRio sent out a message that FrcUserProgram.out had a stack dump."

All I've herd is the "Your robot disconnected, but the lights on the bridge are still blinking" or "The light for your robots disable light was flashing every now and then I don't know why"
Later investigation led to the user watchdog being to aggressively set and a image processing task was taking a little long to complete and the robot ran despite the disable light flashing.

[pitzoid]

Originally Posted by Kingofl337

Pitzoid, if a log of each match with of all of this information is available the field staff doesn't know how to get at it, because every time a robot stops running they usually have no clue idea why.

Well, the issue you run into with training some volunteers is that they'll shake their heads "yes" that they understand everything you've just told them but didn't listen to a word. And some volunteers are just above awesome in how well they handle things

We try to make things like the robot match logs (which by the way was never easily available from IFI) as intuitive as possible, its in the training, why they don't get it I don't know the isolated reason.

BUT, in Match Review, in FMS, when a match is completed (i.e. commit score has been pressed for that match) the robots listed for that Match will turn into either a blue or a red button (depending on alliance, if the match is "not completed", the bots are just listed on a white background), press it (in Match Review) and it pops up the robot telemetry log for the match. The log only records while the match timer is running.

Bots can drop for a number of reasons during a match, hard to say any 5 things are the possible reason. Could have to do with bad code (seen lots of stops in code), low batteries, loose connections, radio placement, the Static discharge issues, etc.... The log does record the FMS connections status alongside the robot telemetry, I've yet to find data that shows FMS connections dropping when robots did (i.e. how we isolated some of the DS issues so quickly), all field equipment runs QOS logs. Telemetry now is still somewhat limited to just communication stuff. I had a talk with some NI folk the other day about bringing out data on every resource a team is using on the cRios and then figuring out a way to distribute this to teams post events. We have big plans for FMS, time and resources will tell what comes to realization.

I think this thread is flying off the track now, if you guys want to start an FMS technical thread we can, we talked about one in the official FIRST forum, but I don't know if anyone would pay attention to it there

[Kingofl337]

Well, how about the ability to dump that data to a thumb drive via a simple click. If a robot stops working a team can request field data review the data back at the pits?

edit:

I think all teams want is to be able trouble shoot issues they encounter on the field. If a team spends the money to goto Atlanta and their robot stops and the answer is "I don't really know" that will be unfortunate.

Also, with the IFI system all that was needed was packet loss and battery voltage. Plus, they sent a rep to every event that knew the system inside and out.

[pitzoid]

Originally Posted by Kingofl337

Well, how about the ability to dump that data to a thumb drive via a simple click. If a robot stops working a team can request field data review the data back at the pits?

Thats a good idea... I can think of even better ways

Maybe next year we can have a way to do this, the hurdle with FIRST is always resources. And to be honest, with the economy going the way it is, who knows what will happen next year? We're all facing economic challenges.

But, remember why its important to keep the matches running on schedule, regional committees bring in officials to do speeches and such (which many pay for the event), and certain venues will have monetary penalties if the event runs late, so there are many many tradeoffs....

I'm all for providing teams more resources, just have to figure a way to get them paid for.

Everyone write Obama and tell him FIRST needs stimulus