Team Fusion #364, Bayou Regional, FMS Woes
 

I'm not sure how many people have had issues at the Bayou Regional, but we haven't been able to have a single good match at the Bayou where we haven't disconnected from FMS. They verified that our router remains connected, but they are saying our cRIO keeps dropping off.

We have replaced everything, and I mean everything regarding the control system. Electrically we're good. Our best guess is that the cRIO is crashing mid match.

Here's the thing we cannot get past. Our code works perfectly tethered and on the practice field using wireless. We can beat the heck out of our robot, and it remains connected. CPU usage hovers around 40%, memory around 40%, we have tons of flash space free. When we put our robot on the field, we're always the last to connect to the FMS, we've had it drop and reconnect multiple times while the field is being setup. The field management bypassed us one match and told us we couldn't play because they couldn't keep a connection.

My question is why? I know we weren't the only ones with the issue today. Kryptonite had the same problem during the same match we were in one time. Another team came to talk to FTA after the award ceremony about the same issue. FTA still claims that they don't see any issues with their system. Maybe they don't see any, but if our code runs perfectly everywhere but the field, that makes me believe there is a problem on the side of the code we cannot touch (cRIO Image, and FMS).

We can sometimes make it through autonomous, and during the match, our robot will come in and out of communication. We finally remained connected for at least half match in our last match of the day, and we did great.

I hate to see our kids' work that they spent 6 hard weeks working on go to waste because of a dumb bug or glitch with the cRIO image.

I know there was a team update on 'lightening' code, but we're not running anything more complicated that what we ran in 2008 on a PIC18 microcontroller, which is way less powerful than the cRIO we're given today. We tried lightening the code during the day between matches, and we finally finished right at 7PM today. We have one match left, and it's the only match we can try to redeem ourselves. We have no closed loop control, and I'm calling it our dumb code, because all it does is take joystick output and apply it to motors. Nothing neat, no safety interlocks, no special features. All of the hard work we worked on is useless now because of a problem that we cannot diagnose outside of the FMS system.

Honestly, I'm exhausted. We've spent 2 days now working on this problem. Our CTA, MJ, looked through all of our code, witnessed it working in the pit, and agreed with us that our code works. We have sat in every match so far. We have a very capable robot, and we know it. It's so hard to accept that we basically missed every match this year for a 12 year old team.

Thanks,
Ryan N

I would also like to add that I have over a year of professional LabVIEW experience. Our code can be found here: http://www.chiefdelphi.com/forums/sh...d.php?t=103902

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?

Is your radio mounted near motors, and anything else that could cause interference?

Is it buried deep within the robot, possibly shielded from the wireless signal?

Was the router used a lot last year? A bit beaten up? Could any foreign material have fallen into it? Could it just be dirty inside?

Is the power connector to the radio nice and tight? Hot glue, rubber bands or velcro straps could help here.

These are all a bit of a reach, but at this point, I'm sure you're willing to try anything!

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Dear Ryan

First off, I'm really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.

I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.

We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven't tried it, although I'm sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year! ;)

Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I'll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn't something to do with the crio rebooting. We did notice that we were dropping packets.

The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we'll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.

Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.

Once again, I'm really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Is the router clear of motors and things that might interfere with it? We had to move ours because FMS didn't like the fact we put it low, near the drive motors.
Try a new cRIO? Image it?
Not really sure what it could be just throwing out some ideas. Issues like this always suck, I remember FMS costing us a match or two last year.
Good luck to your team, and I hope you guys get it sorted.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Our radio is connected below our shooter motors, but we're losing connection while disabled as well, so this is not caused by EMF.

I will update tomorrow after our match if we run or not. I have a bad feeling that it has been the code the entire time. No reason the code is failing, just that it's too complicated for the FMS to handle for some dumb reason. I know it doesn't make sense, but there's nothing else I can think off as to why it works great everywhere except the field.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'm also at a loss for what could be the cause of your communications issue. I'd be curoious though if you're able to replicate the same issue in the pits while tethered and running FMS light?

you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx

direct link to the download: http://208.82.132.46/download/fmslight.setup.msi

i've had good luck with this version on windows 7 but not some of the others... i don't know if it will exhibit the same issues when tethered and with this version of FMS, but it might be a worthwhile test... (quick setup = install, set your ether ip to 10.0.0.5, open FMS, click the big reset button(seems to need it every time you open it fresh), then put your team number on one spot and bypass the others, DS should show FMS control and you should be able to start the match from the FMS panel)

if by chance you ARE able to replicate with FMS tethered in your pit, then immediately try a simple example code if that has issues then it should be something electrical/physical still... if that code is fine but your own code is not - you know where the problem must lie...

hope that helps - good luck and keep us posted.
- Jon

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Make sure your not getting something called a watchdog error. There is a way to disable it in the code. I am not by any means a programmer but this has happened to my team before. The symptoms you describe are pretty accurate to what was happening with us. Good luck.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I've been seeing these symptoms alot, each comes with a smorgasbord of possible causes, and any number of eventual outcomes. There were many when the system was new to us in '09, but there seems to be an increase this year.

One thing I've seen in common is that the electronics were mounted to polycarb sheet. Both '09 and this year the robots are like vandergraph generators (fast moving belts within an isolated shell). Is it possible for a polycarb surface to build up static until the conditions are ripe for it to discharge?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

It sounds like we might be having a similar issue. We've extensively replaced components and code, without a solution (yet). http://www.chiefdelphi.com/forums/sh...hreadid=104713

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Can I get DS log files from each of you emailed to me. Also can you include a description of the cRIO, the DLink radio particulars including the firmware version. I have your code Ryan. I'm not sure if that is having any impact.

I was at a week one and two regional and haven't seen this. Comms issues have typically been firewall, IP, or electrical issues. I've also seen the issues go away when the radio was reprogrammed or swapped out. I wish I knew what was causing those.

PM me for my gmail account, though you can probably guess what it is.

Greg McKaskle

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We've tried 3 radios, all rev A. Two with firmware version 1.21 (one was ours and one from spare parts) and one with firmware version 1.4 (purchased last month). We've tried 2 cRios, both have been 4 slot cRios (no room for 8 slot).

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

At Rutgers our team, along with 75 and 41 all had a similar problem to that described above, except that none of us could connect to the field at all.

Long story short, the automatic radio configuration system was not configuring the radio properly. When we configured it manually (with the explicit approval of the FTA) we were good.

Perhaps checking the radio configuration for oddnesses might help?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Do you know what setting was not good?

Greg McKaskle

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

And also, can we get a breakdown of what the exact WORKING settings were when you manually re-configured?

These settings will be a good reference for us in the future.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We had issues like these last year and replacing the driver station laptop fixed it.

We, to this day, are still trying to figure out why.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We ran our robot for 20 minutes in the practice field wirelessly. Made 30/30 shots, and it ran flawlessly. Right in front of one of the FTA. I asked him what he thought and he blamed the robot still.

I'm sorry, we worked for 3 days, completely replaced all the control system, replaced our code with basic drive code. It always worked out of the pit, but never on the field.

I call bullcrap on it being us. I'm an experienced LabVIEW programmer, 4th year computer engineering student, and very knowledgeable in electrical. Our robot is wired and configured properly.

FMS cost us every single match. We never ran a single match without disconnection. We would disconnect at any time. Before the match, during autonomous, or during teleop.

I hope FIRST figures out what's going on, because there is nothing more depressing in FIRST to see your robot go on the field knowing it's not going to work.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Maybe totally unrelated, but 176 was immobile for long stretches in finals 2 and 3 at Montreal. They worked in autonomous, then stopped dead for the rest of finals 2; then they worked for autonomous in finals 3 and stopped again for about 20 s.

One of their alliance partners was also having some trouble moving around.

(I'm not in Montreal, so I don't know if there were other underlying reasons.)

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

When we had the symptom, we could connect to the radio using a browser but there were *no* settings. Noting, like a broken web page. Weird. We hard-reset the radio and could see settings (but they were default, of course). We compared to a good radio and got the WEP code, entered it all manually, and it worked.

We took our second radio, hard reset it, and got it programmed, and it was good, we could see the settings and all. Tried it in our next match, and both radios were OK after that. Go figure.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The links to FMSlight don't seem to be working does anyone have new ones? I would love to test our control system before our event in two weeks.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The difference between the practice field and the competition field is more than just the presence of FMS. The quality of the wireless signal is going to be very different. Minor RFI issues on your robot, such as noisy power or speed controllers mounted too near the D-Link, have the potential to disrupt match communication without being bad enough to show up during practice.

If you have the D-Link mounted adjacent to anything electical, try separating them. I've seen that fix something that sounds like your symptoms on one team. If you get a chance, try looking at the 5v power going to the D-Link using an oscilloscope. Replacing the Power Distribution Board fixed the same symptoms on another team.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Several teams were experiencing communication issues with the field, us included. Ours weren't anything nearly close to team Fusion's but we tried everything, switching Ethernet cables, moving the radio up away from motors. We still had comm issues and brought it up with FTA along with another team and gave them a list of teams that were having the same issues. We never heard back from them but we understood that they were extremely busy. We dropped it because we saw that majority of the teams weren't having problems and just assumed it was us.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I think it has to do with the crowd. Maybe our router is set to a bad channel or something?

FTA let us run our robot Friday night after everyone else left. No one was in the stands, and just a few people were still on the admin table. We ran a full match without any problems at all. We thought we had fixed our problem and packed it up for the night. Came back this morning and we had the first match of the day, and couldn't connect to the field, even while the other 5 robots were off. I think there is too much interference. I'm not sure why we were the only ones affected the entire competition. They just said we had bad network latency and packet loss. We, again, tried 2 different Dlink routers, one was ours, the other was a pit loaner.

So yea, when no one else is near the field, our robot ran fine. With the crowd and everyone working, the robot didn't work.

We were running all PWM, no CAN, no Camera, no closed loop control. Just the basic FRC Robot Framework, with the configuration for our Cypress board, and the basic controls to make our robot work in some fashion. (bare minimum code & it's attached).

What I've ruled out: it's not programming, it's not the cRIO, ethernet, Dlink, power converter, sidecar, NI Modules...

We never replaced our PD board, but replacing it makes no sense. What's the difference with running it wireless at the practice field and running it on the field as far as power is related? Nothing.

Here's our stripped down code. Don't expect anything extravagant at all, it was written & debugged in about an hour, and is meant to be bare minimum.

Attachment 12341

It's not a programming problem. If it was, it wouldn't work in the pit or the practice field, or even the real field when it does work.

It's also not an electrical problem. We stalled our 4 CIMS, got our voltage down to 7V, and no data loss. We shook, dropped, rattled, beat, and kicked our robot during communication, and again, had no loss. We started up our 4 noisy 550 motors for our shooter to full speed, and started our 4 CIMs at full speed, no load to try to create some bad EMI, and again, no loss or problems.

It's not us. I'm not saying it's the field. I think it's an environmental factor, which isn't our fault, but it is the responsibility of the FTA to control environmental factors. They should be sniffing the air for WiFi hotspots... I know there were tons of WiFi networks being shown. Everyone with a smartphone is polling the air. The ultimate question is why just us? Why were we the only team at the Bayou that constantly had communication failures? Is there something the FTA isn't telling us? Do they have equipment on the 10.3.x.x network for some of their stuff? (Would affect Beachbots 330 too). Conspiracy? I have no idea... I know we weren't the only ones having problems, but I do feel we were cheated out of an equal opportunity during the competition.

Team Fusion all left with a sour taste in our mouth, but we'll manage. FTA did not help us at all, and just stuck with blaming our code (attached), and electrical (explained above). All the mentors are on the same page now. We all know it's not us. We were treated equally. FTA was not able to help anyone really with this issue. When other teams had the issue, they just said to check your code and bandwidth as those are the 'common' problems. I'm upset with FIRST about this. I hate to say this, but this was one of the things that the IFI system didn't have. Someone told me today that IFI can hold huge competitions with dozens of robots running at the same time, without any problems, but FIRST cannot run 6 robots at a time. I'm not blaming any single person or organization, but NI & FIRST needs to get this figured out.

We're not blaming any of us on the team. There were many ideas Thursday and Friday, but after today, we're all confident that our robot is not the issue. It was personally hard for me to remain professional through this whole ordeal. A few of our mentors lost their temper, but I understand why. When we spend 3 days replacing, reconfiguring, and fixing our robot and we yield no results, it gets really frustrating.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I would normally agree, but replacing the Power Distribution Board did fix the issue for another team having exactly the problem you describe. Yes, it makes no sense. No, there should not be a difference between practice and a competition match.

Consider this: if it's the one thing you haven't changed, and the problem is still there, you can't rule it out.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Let me say this about Team Fusion.

I was heart broken that your bot could not compete at the regional. I saw it at the meetup that you hosted at your school and was ready to watch it on the field.

But - at the same time - I have never been so impressed at a team. Team Fusion, even while not being able to move across the field, showed more sportsmanship than any other team at the event. The whole team cheered for everything from their own team being on the field to another team winning an award. They showed gracious professionalism and good sportsmanship throughout the game.

You made me proud to be in FIRST robotics. To see a team that can be having so many problems still be excited about what they are doing is rare. To see a team that - even reaching the quarterfinals remove themselves from the game for the sake of their alliance members is beyond words.

I hope that my team, Panthrobotics, will emulate yours each year. I think most teams have a lot they can learn from yours.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

True story. Anyone who watched the Alliance pairings at Bayou knows that Fusion was picking as the 8th seed, and they immediately called in a replacement so they wouldn't hamstring their alliance. Just before Alliance pairings, Fusion came by my pit, explained what they were going to do and told me that 57 (ranked 18th) was third on their pick list. And then asked me if that was okay, and if I'd be okay with being picked to be on an alliance where the captain would immediately step down. I told them I'd be happy to be picked by them and work on that alliance, and I mean it. We were first pick for #5, but it would've been an honor to play on that #8 alliance for Fusion.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Still there was no solution to our problem. After having com issues i.e. no driving robot almost every match, changing everything from our crio all wires going to each, the radio, sidecar and all wires to and from each. We even changed the power inverter for the radio, checking all the code settings and then running flawlessly in the pit and wirelesly on the practice field in front of the fta. Which i might add we collected balls with the robot and made 30 out of 32 goals in the top, we drove till we were tired of driving, We were again told there was something wrong with our robot rather than a field issue. On top of it all we were ranked 8th as a result of amazing alliances. We ended up picking deserving teams rather than the next best and dropped out as a result of a lack thereof a diagnosis to our problem.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We also had communication problems at the Bayou Regional, with similar symptoms as 364 for our last 2 matches on Friday. We had run all day with no issues, and then all of a sudden we would only work during autonomous modes. We talked to the FTA guy and they told us we had lags greater than 500ms and we were dropping 5000 packets during our matches throughout the day. I believe the average for other teams was less than 20 msec lag and less than 50 dropped packets per match.

We were frustrated that our issues just happened out of nowhere after having an undefeated day. It was at this time that we actually started to hear of 364's similar issues. They seemed to have similar symptoms, and we were one of the teams after the day Friday connecting to the FMS to test our system. Everything worked wireless before we bagged and everything worked tethered. It was only at the regional that we had issues. Overnight I was watching this thread and we did some of the recommendations on here. We got back to working Saturday, but we are not exactly sure which of our numerous changes fixed our issue. We didn't have another bridge or cRio to try, but here is what we did do:

1) Ground tested our system to make sure no electricity was being shorted to the chassis.
2) During ground test, discovered one of our banebot 775's had case shorted sometime after inspection. We replaced it with a new one. (And for future applications we have pretty much decided to go FP motors over the 775)
3) Moved our router/bridge to another location on the robot away from all electrical components and motors.
4) Discovered we had our classmate wifi turned on and turned it off
5) Instituted a complete shutdown of the Classmate after each match to ensure nothing was being held in memory and that our smartdashboard wasn't memory leaking us into oblivion
6) Reduced the resolution on our camera to the lowest setting and dropped the framerate to 15 per second
7) Went back over our C++ code and stripped it down, put alot of our smart stuff like dashboard updates and system polling into a slow do loop that only happened 5 times a second. We put everything but the basic drive functions into that slow loop.
8) Made sure the router is on bridge position before match starts in case it shook to another position
9) Did a lot of praying overnight

We had talked to 364 and they had already done all this with their system so we weren't feeling good coming into Saturday morning about our chances. We however were lucky in that one of those things or some combination fixed our robot enough to be able to finish playing the regional. Our lag dropped down to a bad but acceptable 80 msec average and our packets dropped to 100 average.

We are sorry that 364 had bad issues and could never get their robot to communicate with the field. Please know that the knowledge we gathered from you did help another local team to give the Texas alliance a good run for their money in the finals. Thanks for your gracious professionalism, and I personally hope that your future FIRST outings are more rewarding experiences.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

To get back on topic, I want one specific question answered.

What would make us connect and disconnect before the match even began?

Our configuration for the last match was this:

• Dumbed down code, no CAN, no PID, just connecting joystick outputs to motors and solenoids.
• No camera connected

We had trouble maintaining connection to FMS during the pre-match announcements. Why? No EMI, no vibrations, and we know our power is good from testing all the power and connections in the pit. At this point, all of our hardware had been replaced. Our power test consisted of probing voltages when the robot was still, everything came up clean. We then enabled teleoperated, shook, kicked, dropped, rattled the whole robot, and saw no loss of communication, no dropped packets, and no spikes in latency. I'm confident that the power on our robot is in good shape after when I did to the robot during testing. In no case would our robot be subject to the abuse I gave it.

Even more troubling is why it worked Friday night when no one else was in the venue, and why it didn't work Saturday morning when we were still the only one on the field powered up.

So yea, why would we connect and drop before the match even began? It seriously sounds like a crowded WiFi channel or a resource conflict on the FMS side of things.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan, did you try switching the wireless channel or downloading a utility to see what wireless channels were being used around a field in a typical match? If switching the channel doesn't work then we just need to start building a Faraday cage around the field ;) I hate that you guys had to go through that this year since the robot looked like a mean machine. Are you guys going to be able to attend any other regionals or was that it?

-Brantley
Team Fusion Captain 2007

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Questions for someone who knows more about the FMS than I:

Is there a way to tether a robot directly to the competition field FMS, and bypass wireless altogether?

That could've helped rule out whether this is a wireless issue vs code, etc.

If someone/something created another wireless network with the same "364" SSID, could that cause similar types of problems?

I know usually someone is charged with monitoring for rogue wireless networks though.

I wish I had more suggestions for you...

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Your symptoms sound exactly like ours, except that our problems did not stop when we stayed late on Friday night after everyone else left.

We did monitor the cRIO and radio power with a DMM when the dropouts occurred, and there was no change. However, replacing the power distribution board did fix our problem. I can't explain it, but it worked.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We talked about it, but the FMS is so particular about everything. They have it all setup with a kiosk, and the teams aren't supposed to change anything on the router. We couldn't get FTA to budge on much of anything. The fact that it was connecting sometimes was good enough for them to say that the problem was ours and not theirs.

I wish I would have had software to check how busy the channels were. I feel pretty certain our issue was something to do with conflicting resources on the network. That's the only thing that made sense to me why it worked Friday night when no one was there, and didn't work the next morning.

The fact that it worked fine on Friday night, and that we're using such dumbed down code with no features, just basic drive code leads me to believe that the issue is not code related at all. Also, the fact that it runs everywhere but the playing field lets me know that the problem isn't code related.

We're talking with some people about trying to get pushed into the Lone Star Regional real fast.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Despite our hopes and expectations following a full four and half minutes of connection in the morning, Team 846 was unable to maintain our connection to FMS throughout any of our nine matches.

In the morning, we had a few minutes on the field with FMS to test communications, along with another robot. Though things appeared to magically work following changes to task priority in the code, the same (deployed) code did not work in the actual match with six robots on the field.

It is probably useful to note that, according to FMS logs, less than 10% of matches at NYR ran without at least one robot dropping communications.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We had a few minor latency issues with the Bayou FMS on Friday morning but not after that. Team Fusion had a great robot. It is a travesty they did not get to play! They were definitely robbed.

I agree with the Fusion mentors on this thread, interference of some kind makes the most sense. But I have one (pretty weak) troubleshooting comment - was the radio always mounted in the same location? Did I see a statement about it being mounted "under a motor"? Did you try mounting it high and without any noisy active electronics in close proximity? Were the delays in communications several seconds or 1 minute plus?

Though you replaced the radio many times, did you replace the power and ethernet wiring?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I really hope this works out, we would love to have you come to Houston. Your robot looked fantastic and it's travesty we never got to see it at its full potential.

A program I have found that works well for viewing wireless channels and possible interference is called inSSIDer.

I really hope FIRST can find the cause of these communication issues. The power distribution board and the router configuration seem to be the two things that were never changed. I wonder if configuring your setup as a different team number would have resolved the problem.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We asked about another, but temporary number and they said they can't do it. The router was moved to the best location available on the robot. It was located under two 550 motors, but has now moved to an unpopulated area velcroed to some lexan above the crio, digital sidecar, and our solenoids.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

As inexperienced as I am, I still tried to the point of headaches to figure out what the problem could have been for us. I've never been so frustrated in my life, nor more disappointed. However, I felt like the decision we made regarding finals was the best thing we could have done. We'll see what the future holds, what with this being my last year as a student.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I feel for you guys. We've never missed as much of a regional as you did, but we missed the first 1/3 due to control issues in '09, and I know how horrible we felt. I can't imagine being in your situation.

I didn't see it listed anywhere, but I may have missed it: Have you guys gone out with a different driver station and no cypress board to see what happens?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We went out with 3 different laptops as our driver station.

1. The old style classmate that we received in 2010.
2. A Dell M5040 (we personally don't like this one, and it restarted during our first match on Friday due to pending Windows Updates... go figure)
3. Late 2010 MacBook Pro (running Windows 7 natively)

All three laptops are fully capable of running the Driver Station software along with the Dashboard.

We have not tried running without the cypress board. It's an integral part of our robot design. Running without it would hinder our driver from running. The Cypress board shouldn't be the problem.

On Saturday, when we were desperate, we used the Classmate, with the default dashboard, and the dumbed down code that I've mentioned before. It was a stock FRC system.

Joe, we'll try replacing the PD board when we get a chance. As of right now, we're not registered for another regional. We're going to try to register, and as such, we don't want the students messing with the robot until a final decision has been made.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Yep, we had a very similar issue several times on the field at Bayou.

We never had any issues connecting to the field before the match began. BUT, after running autonomous, we'd still be "connected" but the driver station would display "No Robot Code." When this happened the first time during a practice match on Thursday, we waited to see if it would find the code; never did. The FTA suggested that next time we try to reboot the cRIO in the middle of the match and see what happened. So we listened: every time we "lost robot code" on the field (which happened at least 4 times over the course of Friday and Saturday), we would immediately reboot the cRIO. What would happen:

1) Auto works fine
2) Teleop begins: "No Robot Code"
3) Driver immediately reboots cRIO = lose connection with FMS temporarily
4) Regain connection within ~5/10 secs.
5) "No Robot Code" displays again
6) Regain comm ("find" code) with about 20 sec. left in match

So ultimately, reboting the cRIO worked for us (if I can really say that... We were still out most of the match, obviously). Did you guys ever try this?

But you're right, this still doesn't answer the question of why this occurred in the first place. Our losses were more sporradic than yours, so not sure why ours worked most of the time. When we talked to the FTA about this issue during quals, he blamed it on our robot, too (and you know, who knows, it very well may be the robot). When it happened again during elims, FTA said it still looked like FMS was fine; they said there were no issues on their end, and when the code was "lost" after auto, everything was still reading fine in relation to their connection with our bot. Who knows.

Hope someone can figure this out soon.... We're also attending Lone Star in a few weeks.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I don't think this sounds like the same issue based on Ryan's description of 364's problems. It sounds like your code is crashing or sucking up all the CPU.

I would use the Practice match feature on the Driver Station and watch your CPU and memory on the Charts tab of the Driver Station.

I would also look very carefully at all image processing code and make sure you are limiting the number of particles you are processing and catching any exceptions or handling any errors that may be generated including return values of Null from the various vision pieces.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

2168 seemed to have this problem at NYC.

We never had an issue, period. Not for the entire competition until finals. We went 8-0-0, ranked #1, and had the highest teleop points. Our robot works, and we showed that. That's what makes this so unbelievably frustrating.

During our first semi-finals match... all three robots lost communication on the field. 2168 and 329 came back up but 1676 was disabled all match. The blue team did not see such an event. One of our mentors spent 10 minutes with the FMS manager diagnosing what was causing the problem. He said: "There was nothing they would do but it was very evident from the data the FMS collects that all robots were having 200 - 700ms delays."

1676 changed their bridge and Ethernet cable, and 329 deployed new code.
2168 verified wiring and swapped out one of our Jags which seemed to be failed on the drive train*** and the battery that we used during that match was getting way too low ~7V during the match. It was marked as good. (It is now marked as failed). We checked frantically for shorts but our chassis (as it had been all competition) showed infinite resistance to ground and pwr. The battery cable to the andersen connector was noticeably warm to the touch, andersen to the PD board was not however...

***Later we found, during pit testing, that the snap action breaker at that position seemed to be failing open. We have not yet tested the Jag we thought to be failed, it may have been fine, just not receiving power.

ultimately during our second semi-final match we had intermittent comms again. The lift motors were non-operational and since there was a loss of comm issues the drivetrain was not responsive.

Something to note, our team uses victors on the lift (pwm to digital sidecar on slot 2 of 4 slot crio). All other motors are CAN Jaguars connected to 4slot crio via DB9 connection. All 6 Jaguars remained operation during the matches. The victors were apparently not receving commands, however I was on the field right next to our robot and could see the light on the victors were solid, so PWM signals were apparently being sent?

All testing between matches on the side of the field showed all systems being fully functional. No faults with CAN traffic.

After jerking around in semi-2 we brought the robot back to the pit to try to figure out what could have been going wrong. Our operator noted that our button box (psoc3 FIRSTouch IO Module) was acting "weird". Typically in the IO tab on the driver station we should see all 16 Digital IOs pulled high. All indicators were extinguished, and not changing state with button presses, every now and then random ones would flicker on and off. Analog channels appeared to be responding properly to potentiometer changes.
We're using the classmate for the driverstation. It had not been restarted since the semi match ended.

Wiring for the board is all potted in glue, so its unlikely that it would have failed randomly. We checked wiring however and wiggling the usb connection at the psoc seemed to cause a fluttering of indications on the driver station. So we thought we had a wiring problem. I didn't believe we had a random failure of that board (The usb cable is knotted before leaving the box to prevent any damage inside to cabling), so I plugged it into my personal laptop and opened the driver station. Everything worked flawlessly. I plugged it back into the classmate and the same issue presented itself as before.
When the power cable was plugged into the classmate the indicators fluttering seemed to be more erradic which led me to think there was a noise issue on the classmate usb port or the hub we are using.
Restarting classmate (including battery removal) resolved the issue however, and it couldn't be replicated again.

We were able to replicate the field issue (the victors not responding - no ball lift) once in our pit, but over an hour of testing after that revealed everything to be operational.

We've come up with a list of items to check/replace at our next regional. PD board is at the top of my list, but that doesn't explain the weirdness we saw with the PSOC.

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

Hopefully we can shed some light on what's going on here and come to a resolution. Good luck to everyone in fixing the issues on your bots. Keep us updated.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Team 624, I may know what your issue is. I'm still looking into a detailed fix, but I'd like someone from the team, ideally a mentor, to PM me. Sorry if this sounds secretive, but I saw something at WPI, and I just tracked it down Saturday morning. I want to get a few more data points at this point. And no, this isn't causing all field comms issues.

Greg McKaskle

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Greg

I will PM you later this morning. Thanks

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

In 2010, 816 had numerous issues with the PSOC board - we'd randomly die out mid match, lose one side of our drive the next, and then lose the entire drive but retain all of our aux functions the next. Our 'easy solution' was to unplug the PSOC and replace the custom part of the DS with a game pad. After the season we replaced the PSOC with a CCI from eStop and never had any issues with that robot or our 2011 machine: http://estoprobotics.com/estore/inde...d&productId=33

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?

If we do get to Lone Star, we will be swapping out the PD board, because that is the last thing to swap. I'm also loading our stripped down code and going bare minimal for at least our first match for Thursday to verify we're all good to go, then increase the complexity throughout the day.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.

From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

When you guys run wirelessly on the practice field, are you using WPA/WEP? That is the only potential difference between practice and FMS on the robot side. Perhaps the regulated 12V supply to the wireless bridge is affecting encrypted operations...?

I know the radio configuration software has changed since last season, and I suspect on average robots are far more bandwidth hungry than in the past (lots of cameras streaming back to Dashboards). Those might be two other factors causing the failure when other robots are active?

What a heartbreaking thread to read.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I agree on the PD board - it is hard to see how it could be at fault.

I think the FMS or some OS or library bug is at fault. But in the interest of thoroughness - the link on the practice field was much much shorter. And I know when we used the practice field we setup in the exact same orientation every time (of course not the case on the field). This leads me to think it worth it to move the robot's radio higher leaving it relatively unobstructed by metal plates or motors.

HTH

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?

Good luck in the future, however! A glitch like that really sucks...

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Very slim chance considering FMS uses WPA. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WPA keys are randomly generated by FMS at each event so you team's key will change between each event. The WPA key list is not available to teams.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Lucky you, I just wrote an epic email with screenshots of our logs.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We're still reviewing data and trying to piece things together.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

With all the reported trouble with this issue, including our own, there seems to be a problem out there. I have not seen much discussion of results with the rev. B router in competition. Can anyone report the same trouble with a rev. B router? We will be running a rev.B 1522 at St. Louis this weekend (wk 4), so we will report on what we learn.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We've considered it. I'm curious as to the WEP key list. You said it's generated at each event, so it's not recreated each time you reprogram the router.

I talked with the student that reprogrammed our router. Apparently all you have to do is enter in your team number, and that's it?

So another team could simply bring their spare router, tell the kiosk that they're #364, and they would have our encryption key? Messing us up would be as easy as leaving the router powered on in the pit under some jackets or something, right? That would explain why it would have worked after everyone left on Friday. Power to the pits I think was already cut off. It wouldn't jam our signal on the practice field because we're using their network equipment.

I don't want to believe it is malicious, but it's hard to brush off. FTA was completely unwilling to help us out in regards of getting the spare router, and we even asked for a different (temporary) team number, and they refused to do that for us.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan,

First, it's a WPA key. Second, the kiosk only sets up the radio for bridge mode. I don't have one here to verify, but I'm pretty sure switching over to AP mode wouldn't end up with a fully configured AP that your bridge would try to connect to. I could be wrong on that point, though. An identical radio in bridge mode shouldn't cause problems unless it's hooked up to something with a conflicting IP or something. At any rate, I'm really doubtful on that one. It just seems so unlikely that someone would go to that trouble. Incompetence seems unlikely as well, as who would accidentally program a radio with the wrong number and then leave it plugged in? Still, the self-serve WPA kiosk is definitely a gaping security hole in the system. I wouldn't advocate going back to the bad old days of handing out and programming WPA keys by hand, but handing out a team specific password for the kiosk would be a good idea.

Lastly, when you come to Lone Star, would you consider leaving everything as is for a practice match? I'm pretty sure Lone Star isn't getting the Bayou truck, so it'd be a pretty compelling data point if your robot, as is, does/doesn't work on the Lone Star field.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'm doubtful on it as well. We're planning on leaving everything the same until we know what we're doing. We're not going to unbag it until we know if we're not going to Lone Star.

I just spoke with Greg on the phone for a nearly an hour. I sent him the email I sent above, and talked with him about all what was going on.

He doesn't have much to say yet. He is not sure what is causing the packet loss right before autonomous, or the short chunks of data that are going missing, as it's not enough time to mean a power problem. He said a bad ethernet problem could cause it. I don't believe that to be our case as we did pretty extensive testing to be sure that we were electrically good.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'd really like to see FRC return to a 900MHz radio solution.

Consumer-electronics grade Wifi is just too widespread to count on there being little-to-no-interference.

We're all carrying in literally hundreds of 802.11 wifi-enabled devices that talk in the 2.4GHz and 5GHz bands, never mind that most of the venues have at least one of their own.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Can cell phones really use the bandwith on the field? (Via the the request for network usage) Last year we were told to shut off wireless devices. Now more then ever with cellphone usage up tenfold from last year is this the problem? Were we the the unlucky team with a 2.4 or 5 mhz bandwith?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Did you see this post in 330's thread?

Just to throw that out there.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The fact that it is WEP does NOT make it secure... in fact, some people joke that it is easier to crack WEP at someone's houls (and faster) than to get up and ask them. All commercial wifi setups have flaws. For example, I have seen backtrack 5 crack my home wifi (linksys a/b/g router, set to WEP temporarily) in under two hours. Via a laptop. With WPA2, it took 4 (used the WiFi protected setup vulnerability). A sufficiently determined team COULD crack the encryption in a relatively short timeframe. Or, they could use any of the other specialized linux penetration testing distros. And if the WPA list key is not avaliable to teams, how hard would it be to imitate the FMS setup? (change MAC address on router, set up a network, log attempted connections and passwords)

Yes, I seem to have put too much thought into this. I do not intend to be the kind of person who would do this, however.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Whatever the problem is....I just hope that it is fixed IF/when we go to Lonestar...I know that EVERYONE on the team worked our butts off to build what we all agree is the best robot we have ever made.. and it stinks beyond belief to see it all go to waste over a problem that wasn't even our fault.. Personally I am kinda disappointed with FIRST for how unhelpful they were, they above all others should be the prime example of gracious professionalism and this time they fell short.. However, I am very proud of my team mates and how they conducted themselves through all of our issues and I know that even if we are not able to compete and put all of our hard work to good use, that all that hard work was not completely wasted because we all learned A LOT this year, and still had fun in the process despite the hard time we encountered at the regional.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

You might find it enlightening to consider the total amount of time a given team's wireless network is even active at a regional.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

And again, as I understand it...

Why try to crack the WPA or WEP, whatever it is, encryption when you can simply reprogram your router with another teams number?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

But the FMS would detect duplicate IP addresses because I doubt the install script FIRST uses changes the machine address. So if something nefarious happened, I doubt it involved re-programming a router with the same team number (by FIRST) - which means they would have to hack comms to get the WPA keys. And a MITM attack seems too far-fetched for FIRST students to pull off.

Nope, there has to be another reason. We were a few spots down from you in the pits. I wish I new you were having trouble, we could have swapped some parts around and eliminated some causes. It is so unfair that you had an undiagnosable problem like that! Any luck with Greg?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Greg is digging through the data I do have and I'm looking for more. I'm waiting on log files from another team at Bayou. If you have access to the DS computer, I'd really like to compare aspects of the files between teams with no perceivable issues and ones with issues. PM me if you are able to assist.

I'm also collecting log files as I force specific types of failures. I'm not sure if or when it will be known what caused the failures, but I'm learning of other things that would be nice to log and ways to better diagnose if this happens again.

Greg McKaskle

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Greg, forgive my ignorance, but I'll be telling my programming team about this.

Are there some instructions or something I can look up to help them find the log files?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I don't think the firmware's the problem. When we looked on Saturday at the router's information during regional it said v.1.21.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

We attended the Bayou Regional and almost every time we we connected to Blue 3 we lost connection. The computer said we were connected to the FMS but the communication and robot code lights were off. The RSL never stopped blinking. Also once we were connected to Blue 1 and headed in front of the Blue 3 side ( by the ref table) and lost connection. After speaking with officals they stated that our battery dropped below 8 volts before dropping out, but the battery level on the dashboard was steady. Just my 2 cents.
2078 Driver

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The log files are located in:

C:\Users\Public\Documents\FRC\Log Files

You can view them using the Driver Station Log File Viewer located in:

64-bit Operating System
C:\Program Files (x86)\FRC Driver Station

or

32-bit Operating System
C:\Program Files\FRC Driver Station

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

...hmmmm
George Wallace (SPAM'er, Exploding Bacon and Orlando emcee) said something interesting last night at our SPAM meeting. He was telling me that teams were reporting a dead spot on the field (Orlando regional): blue zone, near the scorer's table. Our freshman IT whiz-kid was telling me that, when we walked down to the field for the Finalist medals he took a peak at the field hardware. Apparently there is a cluster of high end routers located at the scorer's table in that corner. I was just talking to one of our CE/EE's, here at Belcan, who did a lot of work in the RF lab at FAU. Locating a group of routers in a cluster can and will cause near-field cancellation.
... yet another tidbit ...
Eric

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

It's a single AP with 6 antennas (if you look closely, there are 3 for 2.4ghz and 3 for 5ghz).

In the range we are talking about (no more than 50' with no solid obstructions), 802.11 should have no problems with signal strength assuming the bridge is mounted well.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Seems to be the answer everyone is getting from the FTA - there shouldn't be a problem, therefore it isn't a problem, therefore it's your problem.

I guess we should just ignore all the data points that multiple, experienced, highly competitive teams are providing, since there shouldn't be a problem.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I've seen this thread mention many reasons other than 802.11 signal strength as the cause of issues. I only mentioned signal strength, in relation to a post questioning possible issues related to the design of the field AP.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I wasn't intending to pick on you Andrew; your post actually included good information related to the construction of the hardware. But it still highlights the issue that "there shouldn't be a problem with signal strength", yet as Eric said, multiple teams were reporting a "dead spot" on the field and mchristina said the same thing about a different regional, so the data says there is some kind of real problem that smells an awful lot like signal strength in some part of the field.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

After mchristina444's post, it reminded me of what George had told me. So I just wanted to throw it out there so others more knowledgeable could add credence or call shenanigans. So thanks to apalrd for that :) Yet, let me ask a different way: Is it possible for there to be wave cancellations in the near field? ...some odd RF effect? Just curious.


I'm with Gary, though. We're not here to pick on anyone, but there is a body of evidence collecting here. I know people like Greg M. are following up and I am grateful for that. Let's see where the data leads.

Eric

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

There's a body of anecdotal randomness...
Engineers measure things...

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Also, after talking to multiple teams at the Bayou Regional, there was more than just my loss of signal by the scores table. Sadly I don't remember specific team numbers. Talking to the FMS representative, I repeatedly used the word "dead zone" because that was my first thought on the field. He shot down the idea immediately. Having the issue in that one area of the spot. He said "Coincidences happen more often than you think".

But one thing did confuse me, why was this only happening to ~5 teams if it was a dead zone in the wifi? Router placement on the robot? etc?

I'm not going to flat out blame this on the FMS but I do believe it is something to look into.
Having this happen more than 3 times in different matches can really bring down your rankings. I experienced that first hand.

2078 Driver

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The biggest idea I have is some sort of interference on the FMS or the 10.3.yy IP or something of that sort. The biggest evidence is the occurance of our problems only when there was a large crowd or devices. What else could the packet loss come from, since it only happens when there's a crowd?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I've asked my team to pull our logs. We didn't have any comm issues during the Friday/Saturday at Bayou. The only problem we had in a quali match was traced back to controller issues in our second match.

Hopefully this helps.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'll pull logs from our DS. We had control issues in two early matches due to a bad CAN network, and at the start of one match I saw our robot connect, run fine for a few seconds, lose comms, reconnect, lose comms and finally reconnect for good. There was one match in elims where we took FOREVER to connect and couldn't align our robot. Much longer than any other bot on the field. This despite the fact the we power up our robot shortly after the previous match ends. We had good comms during all of our matches as far as I know, however, despite simultaneous camera feeds going on the eeePC we use for a DS.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

So true - I hate what happened to the teams in this thread but there is no evidence (yet). Let's say the FMS said "OK, this is our fault". What direct action (not redesign the system) could they take to fix it? We can't tell them what to fix!

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Opportunity had our routers freak out and die. the one in the kit had the issue and it later litterally fried, then we got a new one and after a while of practicing and by the time we got to competition it started doing it so bad it was every match. it worked in the pit but not well on the feild. when we went to the officials spare parts station and got one lended to us, it worked perfectly, a couple other teams had issues with them too, i think it happened to the last few to hook up connections, or it was that a while after use the parts glitched up badly. Our advice is to bring a brand new one to competitions, and maybe a spare if affordable. i did notice that some of the last few teams to hook up wireless connection had issues staying online and connected on the feild. same with the practice feild

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

That wasnt our problem...

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Our router all connections and everything to do with the control system of the robot was replaced atleast twice the issue was never resolved we checked all code and various settings finally we went to data collecting because we had no support or solution from fta that we hadn't done The robot ran wirelessly on the practice field but did not connect reliably to the field

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

By the support of the FIRST community, Gracious Professionalism of FIRST, and the gracious support of Lucien Junken from Robonauts, Team 118, Team Fusion 364 is officially going to the Lone Star Regional!

Some things to mention at this point. When we get to the competition on Thursday, we will be going out to the field early along with some other problematic teams, and test communications. We're not going to change anything with our configuration, including software or hardware. We want to isolate the problem first, then go about some troubleshooting.

At this point, we are leaning towards RF interference. Our router was set to channel 1, but I'm not sure the channel configuration of other robots.

I know channels 1, 6, and 11 are most often used because their RF frequencies do not overlap.

Even though not all teams have problems, FIRST does have a problem.

Take for example this video from the iPhone 4 unveiling a few years ago:
http://www.youtube.com/watch?v=znxQOPFg2mo

The audience was way smaller. The iPhone 4 could not connect at all. To fix this problem, Steve Jobs had to ask the audience to shut off all electronic devices.

FIRST needs to do the same, or use a different technology. Simply using the 5GHz band of Wireless N would help tremendously.

This doesn't answer the question of why just us, but this is a problem at the competition.

For now, FIRST should ask the audience to place their cell phones into airline mode. Students on the field need to just leave their phone in the pit with them.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'm not sure how the crowd's devices would have affected it (I'm not saying they didn't), but I don't think they could have caused an IP conflict. The 10.0.0.0/8 subnet is fairly uncommon for your average user, and they would need to somehow be on the network (the chance of their phone randomly being connected to your SSID with its encryption key are next to none). The only way that there could have been an IP conflict is malicious intent (and I'll talk about the issues with this momentarily). I don't know a lot (nothing really) about RF, but, I suppose that a lot of phones trying to scan might cause interference, but I would think it would be on the 2.4GHz spectrum (which I think the field doesn't even use here in the states, iirc, the 2.4GHz antennas are disabled, but must be inserted anyway). In terms of localized cancellation, I believe the field AP is rated for 16 networks, and is only broadcasting 7, so I don't think that should be a problem.

About malicious intent, the field uses WPA, as does the practice field, WEP is a broken encryption scheme, and is not used by FIRST. The actual field also probably does not have WPS (it isn't a home router after all). Someone talked about just programming a bridge as another team, it would work, but I feel like the problem would be a little less consistent (IP conflicts are strange beasts). That said, MITM is not out of the range of skills of FIRST students, after all, we all know how to use google, but setting up a MITM attack that would survive lots of network disconnects and reconnects would be fairly complicated (ettercap might be able to do it maybe, but I doubt it would work without significant professional training).

I tend to agree that "It should work, therefore it does" however I am aware that the field doesn't always work, and I really hope that you find the source of your issue, I for one will be watching this to see what the result is.

Good Luck,
Oliver

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I think you are thinking off the Custom Control Interface from eStopRobotics. I haven't gotten a chance to use it, so I can't say if it works well or not.

I've had bad experiences with the cypress board in the past and instead we choose to have a student write a custom program for the teensy board. One of our programmers wrote the code to make it emulate a joystick. I don't have it to post yet but we should soon. He said it was only about 20 lines of code and most of the work was already done by other teensy developers. We haven't run this through a competition yet, we go to Dallas next week.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

They do. I have also seen an AirTight sensor on the scoring table, I was curious what it was so I did some googling: http://www.airtightnetworks.com/home...-security.html

It sounds like you were trying to pull a channel setting off of the radio. he client does not choose the channel in a WiFi setup, the AP does meaning that either all teams or every team in a single station would likely use the same channel. Given Andrew's comment of only 3 antennas per frequency it is likely that all 6 SSIDs are broadcast on the same channel meaning all teams are using the same channel.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

There are a couple devices on the scoring table, the AP, is what broadcasts the networks, the Air Tight, is very good at scanning surrounding networks, and can even shut them down (FIRST hq must be consulted before that happens). I believe FIRST actually has an FCC license to operate those. The air tight may be used for other things, but thats what I've been told about it.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Yeah i'd like to comment on my understanding of the field WiFi setup and maybe this will help clear up a few concerns. I am not an expert on the FRC Field, but i have extensive networking knowledge, I have not been involved in the field setup, but i've done a lot of research on it and I do believe I have a good grasp on this, as such - if i am incorrect - please correct me ;-)

Think of it like this, if you walk into starbucks with your laptop and connect to their wifi, you don't choose the channel you are on - the starbucks router does. If you and 5 other buddies all walk into starbucks and connect - you're all on the same single router, on the same single channel all connecting to chiefdelphi at the same time with no issues. The field is essentially the same situation - your robot routers in bridge mode are equivalent to a laptop at starbucks - but they are really a middleman allowing you to connect both a camera and crio to that network - but they are essentially conneting the same way.

Specialties for the FRC Field
1) I believe in most cases the field WILL be configured for 5ghz, all of our routers are capable of 5ghz, the lack of 5ghz on classmates is irrelevant as they are physically connected for the matches.

2) I'm unsure if every team is given an individual WPA password via the kiosk or if they are all the same WPA password, however the kiosk does set the ip address of the router for sure. Separate passwords could allow them to restrict acces a little better, but it can still be done with a single shared key.

3) From what i've gathered - the field will only accept traffic from those teams that are scheduled on the field, so in theory any additional potential traffic from other teams in queue or elsewhere should not be present on the field network during a match.

4) all teams while connected to the field are essentially part of one big network, in theory you could all connect to each other, and you are all able to connect to the field FMS Server, kinect kiosks, etc... However this traffic COULD easily be limited on the networking side, but i'm unaware if the FRC Field network sets up anything to segment the team networks.

5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...

If i left any details out, let me know... but i believe that's a fairly good representation of the field network.

As for the actual problems that teams experienced and possible causes, i wish i could add some insight to that, i've been racking my brain on it for the past couple of days, watching tirelessly for someone to come up with the magical answer so i can stop wondering as well :-)

Good luck to everyone!
- Jon

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."

Do you have any suggestions of what we can measure Mark?
Eric

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

There's an SSID for each team on the field set to the team number (e.g. "2046"). You can see this with a laptop with 5GHz capabilities, or by logging into your bridge and looking at the settings.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I don't mean to be mean, however some of that is incorrect I will go through the items and comment about my understanding.

1) I agree
2) Each team has a separate WPA key, they are generated Wednesday night I believe
3) This is done by having a separate SSIDs (unbroadcast I _think_, but the same as the team number I would guess) and keys for each team
4) Each team has it's own VLAN, I'm not too good at VLANs so I can't describe this too well, FMS can talk to anything, but each DS cannot talk to other DSs or other Robots (this is why, for those of you that have done field setup, each player station has a labeled port on the SCC, instead of just plugging into the switch)
5) The field AP is reconfigured every match, with the SSIDs for the teams in that match, with the appropriate WPA keys. This is why the scorekeepers must "prestart" the match. The SSIDs are not broadcast I _think_

What you described seems almost like it could be a FMS Light setup for a competition.
I know these things from doing quite a bit of volunteering, and asking the FTA a lot of questions while I do.

I agree, though with that last bit.

- Oliver

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I have all of Team Combustion's logs from Bayou. I have close relations with them since one of their mentors was my mentor at the NASA Stennis Space Center for four internships. We're a sister team of theirs.

I'm focusing (read: trying to focus) on my Advanced Circuits exam I have tomorrow morning at 8AM. After the exam, I'll review their data, try to match up some data with ours and do some comparisons. I'm really interested in seeing what the data will show.

I have send the logs to Greg as well.

I'm still sticking with too much WiFi traffic and interference for our connection problem until I figure out otherwise.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I'd have to disagree with you semantically here Mark. If team A loses comms or their cRIO resets during match 1 at a certain location, that's a data point. If they don't during match 2 at the same location, that's another data point. If team B loses comms at a different location, that's another data point..... specific events and frequency of occurence is data. Pass/fail tests are data as well - it always worked when we did this. The problem is, engineers tend to ignore things that can't be quantified (qualitative data). If a jet engine on the test stand starts spitting parts out the back, it doesn't really matter whether the sensors indicated something went wrong or not - the evidence speaks for itself.

The FMS' job is to communicate with the robots, and it isn't happening hundreds or thousands of times per season. Are the vast majority of the problems robot issues? Probably, but multiple teams have done extensive troubleshooting and don't have issues when they're not running through the FMS. If it's performing as designed, then it's designed wrong. If it has some limitations, either they are not being communicated well to the teams or no one is interested in looking for them.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I think Mark's point is that measurements result in data, numbers, etc. They are independently verifiable and repeatable. Anecdotal data is valuable too, but you need to apply valid statistics to it and work hard to remove bias. Engineers act on hunches and anecdotal data all the time, but that doesn't make the data they acted on to be measurements.

If a team member walks to the FTA and points out that five of the field lights are not working, that is an easily verifiable, quantitative measurement. We all have sensors/eyes for making that measurement. We also have experience helping us to evaluate the impact -- are the clustered or spread, where are they pointing, do we have spares, which should we replace to make the most impact, etc.

If humans had RF receptors, comments about a dead spot would probably be acted upon similarly. But since we don't, I'd hope the FTA would scale the reports by a credibility factor and file it into the "something to watch for" bin. I suspect that if they also saw consistent secondary evidence of a dead spot, they would call Manchester and see how they can make a hard measurement, or verify their hunch, or switch to an alternate setup recommended by FIRST.

RF is complex and not intuitive to those not specifically trained in it -- including myself. Wifi builds on RF, and adds its own 30 years of protocol complexity. The robots have their complexity as well, and as I've said before, the majority of the time, the issues can be demonstrated to be a robot or DS issue.

In the best cases, lets call them category-one, we can make primary and repeatable measurements that everyone can learn from. We can measure the battery to determine if it was at fault. We can jiggle wires and show a particular wire that causes the issue -- because I can reproducibly cause the fault and we can all see it. We could also scope it if necessary and measure the circuit fault and give numerical data to the circuit fault, but that usually isn't necessary. Problems like these result in good learning opportunities.

Other times, category-two, we can only eliminate and verify that common issues aren't present. We replace components until the lack of a fault gives us a likelihood that we have identified the issue. These are far less satisfying and prone to misinterpretation. Often we replace components and the issues doesn't go away. Does that mean the component that was removed was good, or simply that it wasn't the only bad component. In a system where multiple elements can fail and the symptoms look similar, this can be frustrating, and it is hard to make it into a good learning opportunity. Whenever possible, direct and independent measurements of the simplified system will help turn this into the former case where we are all learning more and more satisfied with the outcome.

I believe 364 did a very good job trying to troubleshoot their situation. I wasn't at the event, but I believe that both they AND the FTA and FIRST staff were drawing conclusions based on an educated interpretation of sketchy data. I am trying to be objective and glean what I can from the data. I am reviewing their logs, the logs of other teams at the event, and making my own logs of known failure points to try and identify a set of likely causes and eliminate others. If nothing else, I would like to put the right tools in place so that more of the category-two issues turn into category-one issues. If other teams have logs they would like to send me, PM and I'll give you an email destination.

Greg McKaskle

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I can confirm that the SSIDs are broadcast, and that they are named with the team number; the first competition I went to this year with my laptop I looked at the wifi networks and immediately went "someone is going to get yelled at for connecting over wifi..." After a match or two, I saw that the SSIDs were those of the ones currently in a match, and started to ignore it. I don't know whether they were 5ghz though, as I wasn't about to open a wifi diagnostics tool just to see what network they were on. I do know I have both frequency sets, though- 3x3 wireless a/b/g/n card

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

You're not being mean, I asked you to correct me if i was wrong :-)

I'm just happy to have more detailed information about the FMS specifics, thank you for that. It's actually architected a bit better than I expected which is good to hear ;-) Thanks to everyone that helped clear up my incorrect details.

So just to clarify my points from earlier on the channels - we still have no control over channels, that is going to be assigned by the FMS router, however if we are all on different SSID's they could be all the same channel or all on different channels. If they're on different channels, how do they allocate the channels? at random during pre-start? are they pre-set for each station i.e. Red-1 is always 5ghz channel 11, or something like that? I still don't see how that could cause any of the issues 364 saw, unless they were always on the same channel and it was a noisy channel... It would be good to know how they allocate channels.

- Jon