The communication tides are shifting...
 

After today's lack-luster performance from the FMS on the Einstein field, many people are beginning to lose faith in the FMS and current mode of communication. I happen to be one of them. Problems began at Regionals, and were exacerbated on Einstein. Common sense says that if a robot and DS ran fine in Match 1, if nothing changed between Match 1 and Match 2, the robot should preform just as well in Match 2. Unfortunately, this was not the case.

It might be time for FIRST to shift from their current setup to a more robust and reliable way for robots to talk with the field and DS's.

What say you?

Re: The communication tides are shifting...
 

Never should have moved away from the IFI system.

The current cRIO is a solution akin to using a sledgehammer to drive a finishing nail. Sure, it works, but its clunky.

Re: The communication tides are shifting...
 

Wasn't the cRIO initially designed to be used in an industrial setting as a data logger?

Re: The communication tides are shifting...
 

I think the idea of using Wi-Fi is mostly fine, but the hardware we're using for it (d-link consumer products) isn't the most robust ever. I personally think the cRIO is a godsend over the old PIC18 stuff.

Re: The communication tides are shifting...
 

If they really want to stick with a TCP/IP based solution, they should run it on the licensed 3GHz band, using gear like the stuff made by Ubiquiti Networks.

Re: The communication tides are shifting...
 

It's a real time industrial control system. It's plenty hardy enough for what we're doing and the treatment we're giving it. I'm seriously doubting this is a hardware problem with the cRIO. It's possible it's a software problem with the cRIO or FMS, but the most likely answer really seems to revolve around our consumer-grade WiFi radios. The fact that a firmware change on the router causes massive communication problems is pretty troubling, really.

Re: The communication tides are shifting...
 

/sarcasm/ Yes, let's get rid of the crio and never again using vision processing or any of the cool advanced things introduced with it. /sarcasm/

The crio isn't the problem. It's the connection with the field.

I'm all for a new piece of equipment that better connects to field, but first we have to find one. Something we implement in our team is "don't shoot down an idea unless you have a better one".

So, get your heads a cracken and those brains a moven. Let's find us a new bridge.

Re: The communication tides are shifting...
 

Exactly. We're having data processed on an industrial-grade controller with a consumer-grade transmitter. That's the problem I see. I think the best we can hope for right now is that next year in the KOP we get a nice new radio that's much much more powerful than the one we currently have.

Re: The communication tides are shifting...
 

If the issue actually does stem from our radios, why was this issue not uncovered in 2009 when FIRST switched over to TCP/IP?

Re: The communication tides are shifting...
 

Not sure how your team is doing vision processing... Most teams use the laptop for that to not reduce robot performance... Also, there was vision processing before the C-RIO. Look at Rack 'n Roll (2007)...

Anyways, C-RIO needs to stay. The kit needs some updates with more safety protection (built in) for the thousands of dollars in electronics we have. WI-FI will stay. It's secure and quick. The current problems lie within the field system. Also, the paranoia about wireless work in the pits needs to stop. If the system can't handle wireless pit work, then they are doing something wrong...

Re: The communication tides are shifting...
 

Well, aside from the fact that different wireless bridges were used in 2009, the problem may represent a more complete usage of the platform by teams now than there was then.

Re: The communication tides are shifting...
 

a new model came out this year. so they obviously activated the code to break old ones so everyone byes new ones. :p


a team by us had random communication issues from the beginning of the year. They tested the bridge on two separate robots, made a thread here asking for advice, but nothing stopped it. the communication seemed to drop out randomly

Re: The communication tides are shifting...
 

We actually lost in CT that year to comms issues only uncovered well after the regional, albeit due to irregular coding structure.

In 2010, the radio used was much simpler. The question, really, is what changed between 2011 and 2012 that caused this disaster.

Re: The communication tides are shifting...
 

a few reasons:

1. there were packet size rules, which limited bandwidth, hence why until 2011 nobody really used a live camera feed.

2..they were using different radios (linksys gaming adapters) 2009-2010. 2011 was the introduction of multiport bridges, which allowed the bypass of the cRIO for camera feeds. The reason it became an issue this year might be because the bypassed camera trick wasn't stock in the code in 2011 but was in 2012, resulting in more team using the live camera feeds and in turn increasing the load on the communication equipment.

Re: The communication tides are shifting...
 

I think we need find a radio supplier that will show up at each of the competitions to help us through our troubles.

Also, there needs to be a better way to diagnose the entire system. Event logging in the radios would help.

Re: The communication tides are shifting...
 

The problem with licensed bands is that they are, well, licensed. This adds additional cost and administrative paper work for FIRST. Not to mention the possible inability for teams to use it at their local build site for wireless comms with the robot and the higher cost for radios.

If you want to start pointing fingers, start with the D-Link AP. It is consumer grade hardware not designed for the FIRST application. If the D-Link can be absolved of blame then you start moving to other options.

Re: The communication tides are shifting...
 

Where you see a problem, I see a solution:

The problem: interference from the thousands of 802.11abgn devices we bring into the stadiums with us.

Solution: make sure the robots are on a different band.

The licensed 3GHz band, being licensed, is unlikely to have much, if any interference.

Re: The communication tides are shifting...
 

I don't disagree that being on a licensed band would minimize potential interference. My only concern would be how it would effect teams in their normal build season operation whether it is wireless communications or increased costs. I'm no expert on what we would or wouldn't be allowed to do on a licensed band, but I know there would be a few hurdles to doing it.

Re: The communication tides are shifting...
 

Three ways I can see you handle it.

1) FIRST gets an FCC license at each regional location, and has 3GHz radios for the teams to put on their robots, keep 2.4/5Ghz at home.

2) FIRST gets the fields licensed as mobile transmitters (think TV News Vans), and has 3GHz radios for teams to put on their robots at the events. Use current 2.4/5GHz systems at home.

or

3) Each team gets licensed as a mobile 3GHz transmitter (WAY more paperwork)

Re: The communication tides are shifting...
 

Is it a huge problem to allow teams to build with a D-link but compete with a licensed field supplied router? It wasn't that long ago that we had to share colored flags and give them back to field officials when the match was over. I don't see much difference with the router. The router is basically garbage in garbage out and you would just plug in your cRio and cameras to the field supplied router.

Re: The communication tides are shifting...
 

Or better yet, give the teams 5 port switches and a 1 port wifi adapter... that way all that is swapped is the switch.

Re: The communication tides are shifting...
 

While the commercial grade of the robot radio is definitely a concern, I see a major root cause of the comm issues being the simple fact that all of the data passes through a single point of failure: the field itself. If each team had its own DS radio (provided in the KOP), operating on the field would be no different than operating at home (or ideally, in the pits as well..). This would eliminate the "comm works in the pit, but not on the field" issue.

However, the issue is how to implement this, as the FMS still needs a way to talk to each DS, and laptops only have a single Ethernet port. I propose that a custom DS radio be provided to teams (much like the IFI system did). It would have two ethernet ports, labeled "DS" and "FIELD", but not directly bridge at an Ethernet or IP level between the two ports. Instead, firmware on the radio would only pass to/from the field port key control and logging information, ignoring all other traffic (e.g. camera data) on the DS-robot ethernet/wireless connection.

Conceptually this could be implemented with custom firmware on a commercial home router (with the isolated "internet" port being the field port), although I would hope for competition an industrial-grade solution would be provided.

If so desired, this could be combined with an industrial solution on the robot radio side as well. I definitely hope that any future robot radio solution continue to include 4 ethernet ports; this has been a major enabler to doing more advanced things with the control system.

EDIT: seems like other folks are thinking along the same lines.. I just didn't post fast enough!

Re: The communication tides are shifting...
 

I see some problems with any setup. However, the big one I see here is this:

The system components may share protocols, but they were never designed to work together.

Before you say, "He doesn't know what he's talking about", hear me out. Protocols are a good way to get stuff talking with each other. However, stuff can easily get lost in translation. Ask someone who did surfacing in CAD some little time ago--it wasn't uncommon for a surface to jump if you changed systems. It's less common now, with standards like IGES and STEP around, but it still crops up occasionally. The cRIO and the router weren't designed to work together--they can both use Ethernet, but that doesn't necessarily mean that stuff won't get lost in translation, especially if a connection shakes loose. See "router wasn't designed for the impact of one 150# robot into another".

Field-supplied communication stuff would be a pain to work with. First, you have to have 3 sets of 6. One on the field, one being removed after a match, and one being loaded before a match. You might need a fourth--being set to work in two matches. With the flags, it wasn't bad because there was a specific interface, and no other connection was needed, just place and play. For a router, you have to have a minimum of three connections. First, a secure mounting for the device. Second, a reliable power supply running to the mounting. Third, a communication line running to the mounting. All three have to be standardized and work. Depending on the setup, the router may need to get a new encryption/IP every match, which takes more time, hence my suggestion of a fourth set.


Personally, I'd like to see a brand-new integrated system designed from the ground up for FRC use. That's what the IFI system was when it came along. By "integrated" I mean that every part is designed to work with every other part it has a direct connection with, interchangeably if possible (see: IFI radios). Oh, and designed to handle the forces that FRC bots take when they hit each other--metal on metal.

Re: The communication tides are shifting...
 

On Newton, Team 1717 died once in a qualification match on Red 1, then 3310 and another team died in Red 1 and Red 2 3 matches later. Then in all of the elimination matches Team 1717 played in Red 1, (all but one) they died and weren't allowed any replays. Then in Einstein everyone saw the field fail. And apparently all the other fields had at least one issue or another but were told that it was their robot and not the field. I think that it is time to figure out a better way of communication between the robots and the field, because that is a sad way to finish out the World Championships.

Re: The communication tides are shifting...
 

While the NI equipment was designed for heavy lifting (except the Sidecar, which I compare to being as strong as a wet piece of newspaper in a hurricane) the routers have always been mucking it up. Every single part in the KOP except for the computer and router are things you can't find under the "home and small business" tab at Best Buy (obvious exaggeration, but you get it). In a building full of techies, there will be smartphones and hotspots you cannot stop, and they will be causing interference that is utterly unpredictable and can cause a whole host of errors.

I can't provide you with a grand solution at 1am, but I do know that we have gotten all kinds of positive feedback from NI; I feel like they are working to make the best products for our robots. In the case of the routers, it feels like FIRST struck a deal with a mediocre product and moved on with it. I hope and imagine that isn't the case, but it feels like it. I don't think D-Link feels pressured in the slightest to generate a solution.

Re: The communication tides are shifting...
 

Here's the deal. You are running a 1200 dollar robot controller with a 1000 dollar software (labview). You have a field being run by a several hundred dollar PLC with 1000 dollar software. The thing the interfaces between the two cost 100 bucks. I think that says it all.

Re: The communication tides are shifting...
 

I have a plea to make to FIRST and the teams hosting off season events. Let's try to scrupulously document and analyze the problems and see if we can find out what is actually going on to cause all of the dead robots.

I was not at the Championships, but I was watching online when I could and getting constant updates from my team. And I saw some of the same issues happen at Buckeye and Queen City. It happened to us twice at Queen City, both times our driver rebooted the cRIO to restore control of the robot. Looking at what is happening, it seems to me that the most likely culprits are the Dashboard and the FMS software. I am not saying that it could not be something else, but my CCNA training says look for the most likely culprits.

Number one would be physical layer problems. Pre-2011, this was obviously the power supply. Without the voltage regulator a gaming adapter or router could easily be induced to go offline and reset, making robots lose communications for significant fraction of a match. With the voltage regulator that is much less likely.

The next most likely physical layer culprit would be interference. This could be (when we do demos at places with lots of WiFi we have noticed a tendency for robot control lag) a significant factor. But it would seem more likely to be expressed as control lagging rather than lost comms, except that the FMS might see the lag and automatically disable the robot for safety reasons. But I am assuming if this were the case the FTAs would be able to see that this was done. (Maybe this is not warranted, but in helping get the field set up for our off-season event the last two years it seemed to be the case.)

While it is true, as some have said on CD, that TCP/IP is an off-the-shelf technology not designed for FRC, it is an amazingly robust technology that passes quite a bit of data around the world. (Remember what it was originally created for.) It is versatile and powerful. Even those little D-Links on a robot can process an amazing amount of information. And when configured correctly they have no trouble ignoring what they need to ignore and recognizing what they need to recognize.

My CCNA training says to look next for the biggest bandwidth hogs, which are probably camera streams. Here I have an observation. When practicing with a robot at home, with the router in access point mode with the camera stream not being processed by the cRIO, you can view the stream on another computer than the one running the DS software with no lag. But through the DS there can be a decent amount of choppiness and lag in the video stream. If a team is processing images on the driver's station computer, rather than just viewing them, then perhaps the lag might be causing the dashboard software to do something that results in the robot losing comms. This might well be a/the culprit.

We could go on and on speculating, but I think the best hope for a solution is to get lots of eyes to start analyzing. This is where the off-season events come in. There will likely be a lot of bright, experienced people helping at all of the off-season events. Some of them FTAs and some of them not. Which may mean some fresh eyes on the problem. My gut may be wrong, but it telling me that the problem is most likely in the FMS software or the driver's station software.

Re: The communication tides are shifting...
 

I don't know much about the communications part and the FMS, I am just a CAD and Strategy mentor, but FIRST team 1671 The Buchanan Bird Brains have ran 3 years using the C-rio and not having 1 single dead match in 67 matches (not including Thursday where we probably played 10 matches there and also had no Comm issues), through 4 regionals and a national event. So it is hard for me to believe that this comes down to the field and does not have anything to do with the bot itself. Something on the bot must be causing this as well.

Re: The communication tides are shifting...
 

I'm interested in the outcome in all of this. The way it is all described appears to be the same issues we had in Bayou. Connectivity issues without any reason that is simply blamed on the robot.

Re: The communication tides are shifting...
 

It's obvious that something about the robot is causing this. 118 and its alliance mates worked on Newton. 118 was the only Newton team that didn't work on Einstein. Obviously there's something different about 118. Telling a team WHY their robot is having a problem is helpful. Telling a team that their robot is having a problem isn't. The whole problem here is that FTAs are basically saying "Huh, your robot doesn't seem to work on this field." And then dusting their hands and walking off.

No, it's not quite that bad. Yes, there's a large list of suggestions to go through first, but 364, 118, and others have certainly done all those things. Once you're through that list, you're left with a broken robot and zero help or ideas.

Re: The communication tides are shifting...
 

I do have to say a few small points, for what little it is worth. I worked as a Robot Inspector at both Virginia and North Carolina, and at the latter we had one particular culprit, that while it was a brilliant system, the implimentation to begin with was causing lag.

We were looking at the FMS data in realtime and interfacing with the teams to help solve problems (to my knowledge every issue we had was either related to robot code, low batteries due to the power shutdown during the night, and a few lose connections... no FMS issues). The one team that was initially causing lag had three cameras streaming at an extremely high refresh rate. They toned it down to an acceptable rate as soon as they were informed it was causing lag not only with their bot, but with other robots on the field.

These bandwidth issues seemed to cause lag, not any kind of complete and total interference with a single robot (ie. Red1). Feel free to correct me at any point in time, but those were my observations from behind the Pilot's seat.

Re: The communication tides are shifting...
 

I have a contact at Quantenna Communications... For my co-op, I did wifi testing with their demo boards. It took quite a bit of distance to stress their product (on the order of 300-400 feet in a commercial building). I'll tell them they might have an opportunity for this niche...

Re: The communication tides are shifting...
 

The thing that keeps getting me is the fact that in all of the regionals I have been to, all of the feeds I have seen of champs, all the issues that come forward in the latter parts of the competition are robot issues... our troubleshooting at NC proved that! However, early in the game at any of them there are things that needed working out with the FMS... minor quirks.

We are talking about top notch teams here, NASA supported and more! Teams that won nearly every single match, teams that know what they are doing, enough to make it to Einstien in the first place!

In the hundreds, hundreds of matches that FIRST and this FMS system and the D-Links have gone through for all of FIRST... why was this problem not more common? I am nearly positive from the facts before me (what few they may be) that the D-Link is not to blame here. Something on Einstien's hardware had to be off or not optimized.

The only other thing that occurred to me is the correct setup of the routers. They went from one field to another, the same as going from our workshops to competition, or the practice field. If the routers were not configured correctly we would have the same issue.

What it is, I do not know. I am happy that the D-Link/FMS system works as reliably as I have seen it work in the past year... I saw very few issues at VA and NC... but I am severly dissapointed to hear that the Einstien hardware was never even tested (though I would love to be told otherwise).

Sincerely,
Petrie

Re: The communication tides are shifting...
 

My team's robot (3142) has never had connection problems. A lot of you are saying the control system dropped your team's connection multiple times, but did anyone here do FTC last year with Samantha? Rankings in FTC were largely determined by how many times your robot worked, because no one's robot worked more than 80% of the time. You don't realize how well the FRC FCS works.

I think what FRC has to do with the control system next year is give themselves a large amount of headroom on the bandwidth, and allow use of a more expensive, more reliable radio, along with an inexpensive option for all these new Boys and Girls Club teams. FRC could also take a radio and put an alternate firmware on it (dd-wrt), use QoS for FCS messages, and throttle bandwidth on accessory streams. This could also make flashing the radio quicker, since FIRST controls the protocol.

Also, a lot of control problems could be team-related, as impossible as that seems. Going back to FTC (experiencing a similar problem), my team had constant connection issues last year. This year, my team got a new Samantha, a new brick, new code, a new RobotC version, and there was a new FCS. The only shared factor between the two seasons was me, the programmer. My team still had connection issues. It has to be me, somehow, because most other teams had much fewer connection problems. Multiple people looked at my code, and couldn't see a problem. Yes, FTC is not FRC, but parallels can be drawn.

Re: The communication tides are shifting...
 

The point with the cRIO and WiFi connection is that we can get experience with real-world technologies and practices. The IFI controller and its radio were unique to FIRST, so they didn't help us in that way.

If the D-Link or TCP/IP were to be scrapped for something better, what would work well that other people/industries actually use?

I keep thinking of the DIY hobbiest community and its similarities. What would people use if they were making a robot? Arduino ethernet and wifi shields come to mind, with XBee modules. RC cars, airplanes, and helicopters also come to mind, what technology do they use? (I mean pro hobbiests, not $20 RadioShack toys).

And if these are not applicable or too hobbiest oriented, what technology do they use in industry? How do iRobot's military robots communicate? Boston Dynamic's Big Dog? All those quadrotors in development by universities? The ones companies can buy?

Re: The communication tides are shifting...
 

Hobby wise, at least for remote control, I believe Spektrum is the go to for radios/receivers. They use the 2.4GHz ISM-band and some fancy techniques and redundancies for improving reliability. People fly heavy, multi-thousand dollar R/C aircraft with these things so they certainly don't want any loss of communication. But again, this is an apples to oranges comparison as Spektrum controls the protocol from top to bottom.

Re: The communication tides are shifting...
 

I was under the impression, through the rumor mill, that FIRST had intended to deploy OpenWRT on future WiFi hardware. I know from my own experiences building my own robot control system with Parallax Propellers, a NetGear router and DD-WRT that there is more control to be gotten there.

I think it's too soon to assume we need to scrap WiFi hardware. It's probably not to soon to assume we need to scrap software for WiFi routers designed to work in them while they are stationary.

A hacked firmware would open the door to in-access point diagnostics.

Re: The communication tides are shifting...
 

Spektrum, Futaba, JR, and I think there's another brand all use the same 2.4GHz frequency. Each has its own set of hardware, but any Spektrum reciever can bind to any Spektrum transmitter (within a certain amount), and the same for the others. You could have huge flights with everyone using a different setup, and not a plane will go down due to interference. Even with all the transmissions from all those transmitters.

If we didn't need all these sensors and the FMS, I'd say to try using R/C controllers...

Re: The communication tides are shifting...
 

If we just had people from each of the robot communication companies, it would probably help a lot since they would probably know more about their product than we do.

Re: The communication tides are shifting...
 

I think the FMS hardware that resides inside the Scorpion Case has been tested along with the PanelView screens that all of the ref's access before the rig was shipped field components and FMS gear. The field components were brand new but the FMS gear is also from last year. As far as I am aware the concept behind the AP controller getting controls from FMS about match information in the pre-start period and then passing it to the AP has remained the same ever since switching to the bridge setup.

This issue has happened on field all across the nation last year and this year. There are 18 trucks plus the one that resides on FedEx HQ and probably all of them have experienced that field issue.

The thing to do now is to go through all of the field data and FTA logs about which field did what under certain circumstances in certain environments. And I am sure FRC Engineering department will be busy all summer preparing a new fix for next year.

Re: The communication tides are shifting...
 

I think this task may not work for some cases. I can easily see with all the possible issues that some cases are in fact robot issues.

Team 11 had such issues and when the field logs were reviewed all that was discovered was that the robot lost communications.

The troubleshooting process entire simply needs to be expanded as well as the monitoring. If the logs were in fact highly detailed I think that the diligent field personel would have spotted faster ways to put a stop to what happened during Einstein.

In any event I agree this problem must, can and will be solved.

Re: The communication tides are shifting...
 

I think we may be missing the obvious solution.



</troll>

Re: The communication tides are shifting...
 

Begin long and rambling explanation:

I had the chance to talk with an engineer from NI at great length about what goes on inside the field system. Over the course of a few hours, I am fairly sure I asked about every single aspect of the field and how it works (and more importantly, why it doesn't). This is what I was told:

The field itself has two PLC's that handle the scoring hardware and a PLC that handles the bridges. These PLC's talk serial over IP using bridges to connect to a dedicated LAN. These PLC's are only restarted maybe once a regional, and are very resilient. The only time they have to be restarted, is if for some reason the system experiences a power surge that causes one to 'disappear' while it restarts. Next in line on the field are the PanelView systems, which are just win7 tablets with touchscreens running a simple interface to talk with the scoring computer, which is itself a part of the FMS, but that in a minute.

Next in the field is the Robot, the robot has a relatively new cRIO real time controller that is very advanced compared to what FIRST had previously. This system is VERY robust, considering that student programmers are writing code that is for the most part poorly thought out. There are many teams that write good code, but there are many more that don't. The only point at which the code on the machine can easily be blamed with 100% confidence is CAN. The CAN bus system was never designed to work like this, where the machine is expected to do a rapid start up and perform dynamic hand-offs of the CAN master node (the switch from auto to tele). Assuming that CAN is not in use, and that there is good code on the machine, the cRIO is a bulletproof system.

On the other end of the line is of course the DS, the DS is a simple labview program that communicates with the robot to convey the info from the hardware connected to the DS. In my 3.5 years of FRC, I have never seen a problem with the DS. The Dashboard on the other hand, is full of holes. When teams use the smart dashboard, they are asking for trouble, the same goes for all the other dynamic dashboards. These programs are too broad in scope and attempt to fit too wide an application. The only way to write a truly stable dashboard is to write it yourself for your own robot.

So now we are at the physical connections in the system. The IP network is run using Cisco managed hardware that is, for what it's worth, very good equipment. That being said, there are a few bugs. For one, the main WiFi transmitter is just one unit. It is a very special, very powerful unit, but it is still just one transmitter. There is also the issue of it just being Cisco equipment. For that one, some background is needed: when a router is approaching the maximum amount it can handle, it will start dropping packets, TCP which allows for this, will simply request the missing packets at a later time, when there will presumably be less network usage. But here is where the fun part is, how does the router decide where to drop data from? Easy, starting at the lowest numbered numerical address, moving up. This easily explains why old teams (low number) see many more connection problems than the rookies (high number). This is unfortunately just the way the cookie crumbles, so there is no way around it short of coming up with a gigabit WiFi system.

So now for the last part, the FMS. The FMS is a fully redundant system consisting of two rack-mounted servers in the scorpion case. This system is the week link of the entire field. The FMS itself has to handle ALL of the following hardware/software functions:

• 9 VLANS
• 2 PLC's
• 3 Monitors
• 18 dynamic data sources
• A twitter feed (not much here, but it all adds up)
• The sundial system
• WPA encrytion cycling
• Match Scheduling

To top it all off, the whole thing is written in VB.NET! If you want a laugh, head over to AndyMark, where under the field rental section there is a place to download FMS light, which is the same program, just without the bits that handle the Field hardware. You will quickly find that it is only barely stable, and that the it suffers from huge amounts of odd quirks inherent to VB.NET apps. It is clear to me that the FMS program itself is the root of this evil. Since it handles the WiFi system, it is also to blame for the oddness of connections that work perfectly on one field but not on another, or robots that work on the practice field, but not on the real field. The FMS program is in dire need of a re-write, preferably in LabVIEW, which most FTA's can debug errors from very quickly.

Some of you may remember from watching the Alamo regional Finals that there is a very awkward pause during the finals right after one team called for their backup robot. Look it up on Blue Alliance if you haven't seen it, it is one of the better times the judges have had to stall. When I asked later what happened, the Score keeper (who actually sits at the FMS console the majority of the time) said that the system just refused to take the team substitution. It wound up requiring the entire field to be reset and rebooted. This was an issue that had never come up before, but was attributed to a rare access violation within the FMS program.

All in all, we are dealing with a very robust system. The only part of it that is not robust is the FMS program itself. To me, the only solution is to scrap the current VB.NET program entirely, and rewrite it from the ground up in a more sensible language like LabVIEW.

Interestingly enough, we still have our Maize Craze robot in (what we think) is workable condition. We might bring it to the championship pit next year for kicks and giggles.

Re: The communication tides are shifting...
 

Firstly, let me thank you for sharing all that information.

Secondly:
No language insures in the absolute sense you can't have an error at all.

I share some of your lack of enthusiasm personally for .NET languages, however that doesn't mean that I can ignore that a process that produced the buggy product in the first place is prone to be just as buggy on the rebound (pun intended).

If it turns out the that FMS system is quirky they should troubleshoot, repair, replace as necessary.

Re: The communication tides are shifting...
 

MAldridge - thanks for the explanation.

We certainly have had our share of programming errors and loose wires, but have also experienced unexplainable field errors.

From a network prospective, the artificial means of protecting available capacity, by trying to enforce no WiFi access in the arena and the pits, doesn't scale and isn't enforceable - different venues have different levels of control over access points, use of wireless in the audience is increasing (via mobile hotspots and the like), and teams are encouraged to transfer more and more data between the DS and the robot.

I would love to see a transition to another frequency band, but the additional cost is probably not realistic. Regardless, the FMS needs to be robust, both now, and in the future.

Re: The communication tides are shifting...
 

the policing of wifi traffic is what doesn't make sense to me. It is possible to MAC address lock the D-Links to a specific AP, so why aren't they? As near as it was explained to me, the master transmitter doesn't have a single MAC, and there is no way to insure the proper MAC is assigned to the right VLAN every time.

As for a sensible language, I base sensible of of what the guy running it can debug. Since most FTA's do labview, it is sensible that they could troubleshoot it. Also, since the rest of the system is already written in labview, it makes sense to have it all done in one language.

Re: The communication tides are shifting...
 

So I'm a mentor for Team 900. We were at both the Virginia and Raleigh regionals. You guys did an awesome job keeping things in line. It was truly top notch work.

However, I do feel the need to point out that the issue with our robot and the three cameras causing lag for the other robots was not an issue we actually solved in any meaningful way. We were watching our own bandwidth and it remained the same throughout the matches despite us fiddling with settings. I'm not sure what changed but it wasn't our robot despite our best efforts.

I do have a bigger concern with being told that a single robot is somehow taking all the bandwidth away from the other 5 robots on the field. This is a problem that should not be occurring. Quality of Service (QoS) should enable the bandwidth to be shared evenly amongst the teams.

There needs to be more transparency into how the FMS system operates. A complete breakdown of what parts and software are used will enable more eyes for troubleshooting and will hopefully enable teams to host better off season events.

We saw FMS problems at both Virginia and Raleigh. I'm not surprised other teams are having similar issues. I'd also like to echo the sentiment that a better transceiver system needs to be worked out. I'm not sure what that system should look like but the COTS D-Link and Linksys stuff isn't cutting it.

Re: The communication tides are shifting...
 

I was trying really hard not to step into any of these threads, but there is some information that needs to be included in this thread, and I feel compelled to post it.


-Everything between the robots and DS's uses UDP/IP which has none of the retransmission problems TCP has. As packets are sent every 20ms and include a CRC, a missed packet will cause a 20ms lag (not bad at all) and it will ignore the packet if the CRC is bad. The FMS also talks UDP to the DS, and the DS talks UDP to the Dashboard. Only the camera uses TCP for the video stream, which is generally unrelated to completely dropped comm (camera problems will show up as lag caused by late packets, and would be seen by a team from the beginning).

-Prior to one of the matches, I opened a WiFi scanner on my phone. The field uses one 5ghz channel for all 6 networks, and nothing else was using 5ghz for 802.11a/n (b/g only run on 2.4ghz). Cellular networks don't run anywhere near 5ghz (they use 700-800mhz, "PCS" 1900mhz, and a funny "AWS" band that splits uplink/downlink on 1700/2100mhz) and are not the cause of interference (if it is wireless interference, which I don't think it is).

-The field-end AP is a Cisco commercial AP, the same one has been used since the control system was introduced in 2009. It's operating exactly in it's intended environment (fixed AP for multiple virtual networks), and any problems with it would show up with all robots.

- The weak link is certainly the D-link AP. Many teams this season have seen random communication loss at their later events. I'm sure a consumer-grade wireless AP designed to sit on a desk or shelf dosen't like being thrown over a bump at 10 ft/sec hundreds of times.

-I believe that any part that is not industrial or automotive grade, or designed specifically for FIRST will not be robust enough for a FRC robot, especially if it is a single point of failure such as the radio.

-I did download FMS light, and the 2011 version of FMS Delta/Full (Full won't run without the PLC, so my play stopped there), and found that it was clearly written in a .NET language and was not terribly stable. I was able to crash FMS Full quite repeatably simply by not having a PLC for it to talk to.

--At an off-season event in 2010, the circuit powering the field tripped a breaker and shut down the FMS server mid-match. Upon reboot, the FTA's found that it would crash when opening because it had entered a date in the MSSQL table that was invalid, and it would not read the tables at all. As the tables were encrypted, the FTA's literally had to guess the password to manually open the tables in MS SQL and fix the date, and found that the password was a string related to FIRST (I don't remember what it was).

Re: The communication tides are shifting...
 

I'm interested to read more.

Re: The communication tides are shifting...
 

I second that.

Re: The communication tides are shifting...
 

this must be the problem ;o) (conclusion by a prejudiced VxWorks & Linux guy)

Re: The communication tides are shifting...
 

We had one round where we were dead the entire round. A FRC tech person came over and viewed the DS logs and tested the robot. He found nothing unusual. We had a second round where we were dead after autonomous, then disabled the Ethernet port, re-enabled, and it all came back. It took around a minute to do this. Since we were not doing well, it was not a big issue.

It would be great see a detailed network diagram with configuration files of the devices. I understand they need to keep the system secure and not sharing details helps. If there are 9 VLANs (why 9 with 6 robots?), it should be easy to guarantee every team 100Mbs of bandwidth. The cRIO is 100Mbs plus maybe another 1Mbs for the camera stream. What do the 6 bridged radios connect to? I looked once and it wasn't a DLINK, but can't remember what it is. This shuold be a business/enterprise class device. There could also be an issue with the queues on the FMS Ethernet hardware, but if it can't handle max 600Mbs of traffic non-blocking, it must be outdated or mis-configured. Depending on frame sizes, a GigE port can start dropping traffic at 600Mbs depending on the quality of the device. Another possibility would be to use secure tunnels instead of VLANs. An IPSec tunnel with 100Mbs guarantee between the robot and the wireless router with the IP traffic receiving the highest priority marking. If they are only dealing with the Layer 2 VLANs, they have no way of guaranteeing the Layer 3 IP traffic. All traffic should also be mirrored to another port so Wireshark or the like can run an offline trace in case there is an issue or challenge. They should be able to determine what every dropped frame/packet is and where it came from. If dropping radios is a issue, I bet it can be traced back to the same issue in all cases.

I agree with a previous poster that the hundreds if not 1000s of WiFi capable smart phones/ipads/kindles/laptops are always searching for networks. Maybe this is impacting the radios. At the figerlakes (RIT), they actaully swapped out the robot radios with a business class DLINK due to discovered issues with large amounts of wireless traffic and the way the DLINK discovered/connected to these networks. I believe it was related to how many MAC address the cheaper DLINK can store and how long it took it to clear MACs. We do need better radios.

Re: The communication tides are shifting...
 

Hmm... OK, I've read through the majority of the posts in this thread and I'm surprised that we haven't had any RF folks like the hams chime in. So, since I'm a ham operator of almost 40 years experience with antennas, I figure that I might as well open my big mouth and put in my $.02. Essentially, I think that there are several RF-related issues that could/should be taken into account.

The Dlink is a commercial, consumer wireless router designed for static mounting in a relatively benign RF environment. If we could characterize the Dlink's RF transmission pattern, I suspect that we'd find that the DLink is not a true omni-directional antenna. And, we're running it in an mobile, hostile RF environment. High-power motors, weird mounting angles, mounting near the cRio with lots of stray RF fields (unshielded PWM cables, the PDB, digital side car and more) and even mounting inside of signal-blocking structures like aluminum pillars all have the potential for interfering with RF signals. The only requirement for mounting the radio is that the inspector can see the status LEDs. But, the unit can be mounted in such as way that the RF transmission capabilities are seriously degraded while still having the LEDs visible.

There is a reason that most industrial WiFi units have small antennas protruding from them. It's because the little antennas the stick out have a higher RF gain and are more omni-directional than the small patch antennas found in the Dlink unit. In addition, IEEE 802.11n is capable of multi-in/multi-out (MIMO) operation that allows for antenna diversity so the antenna with the strongest signal can be selected and can support switching antennas on an as-needed basis during the match as the robot moves on the field.

So, I think that if you switch routers with ones that support multiple omni-directional antennas like the DIR-615 or DIR-655 as well as having Quality of Service (QoS) support (the ability to prioritize the DS packets and de-emphasize the video if bandwitdh is squeezed), perhaps even to support the remote mounting of the antenna on the center of the robot, that you might be able to address many of the loss of signal issues with the FMS. Additionally, mounting the receiver antennas *over* the center of the field instead of to the sides could also help.

As to licensing a section of the radio spectrum, that's not likely to be economically feasible. Companies like Verizon, Google and others spend literally billions of dollars to lease bandwidth in big government auctions run by the FCC. Even with 3000 new teams from the Boys & Girls clubs paying $5K a pop for registration, it's highly unlikely that FIRST could afford to license a chunk of the bandwidth for use -- let alone convince a radio manufacturer to create radios cheap enough for such a small market.

Of course, the lower the frequency, the the further the distance it will transmit. There are several, open ISM bands that do not require licensing. The 2.4 GHz and 5 GHz WiFi bands are examples. The early serial modems used several years ago where 900 MHz (another ISM band) modems I think. But, the field isn't so big that it justifies dropping to a lower frequency. And, we have to think about countries other than the US as well. The US 2.4 GHz WiFi band overlaps an Israeli military frequency if memory serves me correctly.

So, a dual-band (2.4 GHz/5 GHz) radio with MIMO support and multiple omni-antennas with antenna diversity support and some guidelines for mounting the radio would likely solve most of our FMS connection issues. These types of routers might cost an extra $100 or so more, but the ability to play the game without communications issues would be priceless, IMHO.

My $.02,

Mike

Re: The communication tides are shifting...
 

Why is that?

From what I understand, the current smart dashboard is just sending keys and values over the network to the laptop.

Re: The communication tides are shifting...
 

I consider it to be full of holes because it was not written for a single application. If you think about it for a while, you always come back to the idea that however talented some people are, the software guys on FRC teams are high school students, and the odds that thier code will not screw something up on the dashboard, and that they will configure it properly, are pretty low. We played around with several of the 'commercially available' solutions, such as smart dashboard and zomb, but couldn't get either one to run reliably the whole time. We coded our own and that worked the best.

As for the wireless with the omni antenna, that would be great if it could be acquired cheaply. You are correct about frequency overlap, but I think you have the country wrong (my ARRL handbook is not handy, or I would check). I think though that the problem that you would encounter with an antenna is that then you would have to explain to teams proper cable routing and making sure the feedline was built right. Good luck with that one.

A good radio is a part of the solution, the question is what makes a good radio.

Re: The communication tides are shifting...
 

Just to point out, obviously several companies sell the prequisite reverse polarity SMA connectorized, precabled, and ready to attach antennas typically found on WiFi hardware. I have quite the selection of them I use for network evaluation and wardriving.

Course the problem is that if you vertical mount an omni and the field AP is directly overhead, you're probably in the weakest lobe of the field strength.

Perhaps a biquad mounted flat on the top of the robot on a copper clad PCB, parallel to the surface of the field with the APs directly overhead might be an option.

Re: The communication tides are shifting...
 

I don't want to point fingers at the D-Link directly with this post merely point out a few common thoughts that have been thrown around.

1) Perhaps the biggest problem anyone has in pinning any root cause down is that no two robots are alike. For that matter, not every team is running the same D-Link AP. There are multiple hardware revisions floating around out there and multiple firmware versions as well. Thus is the nature of not having full control of a commercial product. Even the same "version" of a D-Link AP may vary with quality control issues (bad solder joints, loose connections, DOA components) and parts are often substituted in due to obsolescence, cost, and availability issues. One of the first things you learn as an engineer when troubleshooting is to control the variables and only change one thing at a time. FIRST cannot do this with the D-Link as teams can get them from anywhere as long as they are the DAP-1522. Some vendors have old stock with old revisions, some with new stock, etc. Revisions to hardware/software are not made "just because". Usually there is a distinct defect that is being addressed in a revision. Theoretically a DAP-1522 is a DAP-1522 but even a small change could have a disastrous effect in obscure use cases.

2) After watching the webcast yesterday I went off in search of some more obscure info on the D-Link DAP-1522. I had considered this before as the issues throughout the season piqued my interest. The most interesting things I found were some nice pictures of the D-Link's internals. There were three things I noticed:

• Save bad or weak solder joints, nothing about the design or mounting screamed out to me as being horrible with respect to the harsh mechanical environment we subject the D-Link to.
• There are multiple revisions of the hardware out in the wild. See the pictures below. One version does not have the RF connectors glued, another does.
• The u.fl connectors to the antenna seem to be the most likely point of mechanical shock/vibration failures. It appears that this concern would be addressed with the glued hardware. I've never used a u.fl connector, but it does seem like a pretty unreliable physical connection given what these APs may be subject to on the robot. These things are only 2mm tall!

Re: The communication tides are shifting...
 

Also tried VERY hard not to jump into this, but some worrisome ideas are being tossed around...

Our team has observed and experienced multiple FMS failures throughout this season, very similar to what occurred in St. Louis (not just during the finals). It's highly likely that some component(s) of FMS other than robot hardware or software are causing it to fail.

The general technical requirements of FMS (explained in multiple post before, so I won't repeat them) are WELL WITHIN what WiFi + basic TCP/IP networking can handle, with all necessary access controls and bandwidth allocation limits. There is certainly no need to even consider switching to alternative radio technologies (we'll all be wishing to be back on WiFi in no time...:-).

There are two key issues:

1. As far as networking setups go FMS is fairly simple (not trivial, but for a truly competent network architect it is pretty pedestrian). The amount of traffic is also relatively trivial (compared to a commercial environment, not a home router with an XBOX or two). All FMS traffic should be 100% logged for post-mortem analysis. Again, it is well within technical capabilities to do so. The real challenge lies in the availability of people/tools capable of analyzing that data on the spot and diagnosing the problem.

2. Consumer-grade wireless networking devices have NOTORIOUSLY bad firmware. They are built for "easy setup" and "check box features" for marketing purposes, but as anybody who's ever tried any of the more sophisticated features in consumer routers (e.g. web filtering or QoS, not to mention throwing larger numbers of devices at them) knows, these features are often buggy or simply don't work at all. The usage pattern of FMS pushes the D-Link devices into areas they are not really built for and do not experience in normal consumer usage (and are therefore very unlikely to be tested/debugged/fixed - not because they're "bad" products, but because this is not their normal operating regime and their target consumers simply don't care).

It appears that FIRST will have to allocate more resources to both #1 (understanding what's going on through careful trace analysis) and #2 (creating an environment for effectively simulating worst-case game conditions and figuring out what brand of equipment satisfies those requirements or how to configure it to avoid pitfalls). The solution is unlikely to be "build custom hardware or firmware", because statistically it's much more likely to be even buggier than any widely-used commercial solution, not to mention economic (non)viability of that approach.

I suspect that there may be vendors willing to hep out with both #1 and #2, but the biggest challenge may not be technical but human (relinquishing a bit of control and allowing another party to step in).
Broken code is usually easier to fix than bruised egos :-).

Re: The communication tides are shifting...
 

I like the HAMs post:) - using remote antennas mounted away from the other electrical components is probably a good idea. FIRST could provide the same model with the same gain to teams. I use a Hawking antenna at home. Normally, this is a more exspensive radio with the appropriate BNC connectors, but may help. At the RIT event, they used the FIRST approved backup radios with the dual external antennas and I don't remember hearing about any problems. I know we had issues in Cleveland, but not to the extent of St Louis - less people/devices in CLE.

Re: The communication tides are shifting...
 

If widespread communication issues were validated, everything would be invalidated. Our win against your alliance in the semifinals (I don't remember if you guys had any disconnected in those matches). The 548/118 win over us in the finals (we had a disconnected 68 in the first finals).

The Head Ref on Newton seemed...quite unwilling to bend about anything, and much less willing to replay matches than any other ref this season, even in cases where there were blatant faults (ie, scoring display froze mid match).

Re: The communication tides are shifting...
 

Just curious, are the D-Links configured to listen to all the 2.4GHz chatter? If they are, it would certainly seem reasonable to make that not happen!

Re: The communication tides are shifting...
 

They're configured for only 5 GHz.

It appears that the firmware in the DAP-1522 still listens to 2.4 GHz and puts all the detected access points in a list, regardless of its configuration. It won't connect to anything not on 5 GHz, but it only ignores the 2.4 GHz networks after it has already found and recorded them.

Re: The communication tides are shifting...
 

also include Clearwire's WiMAX service in St. Louis. It uses licensed 2.5Ghz spectrum. Anyone with a Sprint EVO/EVO3D/Moto Photon/Nexus-S/Galaxy-S/Epic and using wireless broadband would be in this group.

Re: The communication tides are shifting...
 

Agreed, if for nothing else than to stop the flood of speculation, aspersions and misinformation that I am seeing here. :mad:

I believe that FIRST will indeed find our why there were problems, tell us all, and fix it once and forever. We just need some patience for them to get it right.

Re: The communication tides are shifting...
 

Agreed.

Went to VEX Worlds and saw hundreds of matches run with no dead robots. This was using WIFI and literally had dozens of the robots on at the same time.

IFI only got better after they left FRC. They have a VEX ARM9 that's more than suitable for vision processing and have a decade of robot competition field experience !

IFI - stands for Innovation FIRST International, they were a company founded by FIRST mentors for FIRST teams.

Re: The communication tides are shifting...
 

I wasn't going to jump into the side debate of using VEX or the old IFI system, but here goes. To preface this, I love the VEX system. I have had one of my own since they came out. I have just over a dozen kits of various ages (including four that are less than three months old) in my classroom, and they get used. We have not done VEX competition as a team, but have been to them as a volunteer and a spectator. I really like VEX. And if we used the VEX system some teams would not see a huge difference in their robot capability. And many teams would see an order of magnitude drop. The VEX system, and the old IFI controllers, are not as powerful or versatile as the cRIO.

As someone who spends a good amount of time teaching kids to program robots, with experience with the current FRC control system, the old IFI system, VEX, Tetrix, BOEBots, Ridgesoft robots and a few other systems, I prefer VEX if I am going to teach the kids so that they can quickly create their own robots. And I greatly prefer using the cRIO if the robot is going to have an autonomous mode that will do something meaningful in 15 seconds and have as many motors and sensors as our robot had this year. We tried this fall to get last year's robot to run using VEX controls, and when we got to three microcontrollers in order to make it work I gave up. And the old IFI system I found harder to use and less powerful. Also, I will admit I have never seen anything quite like the finals happen under the old IFI system or at a VEX competition. But I have seen FRC competition delayed for hours when using the old IFI system. Not because of their system per se but because of how the field management was set up. If VEX competitions had 140-150 pound robots racing around at the rate of speed of FRC robots, they would need a more complex field management system to insure the safety of the people and field. And it might well be prone to significantly more problems.

Re: The communication tides are shifting...
 

I emailed my contact at Quantenna communications today. I sent him a link to FIRST's website. He seemed very enthused about the whole thing, and told me he'd present it to his boss.

I'm not sure how companies become suppliers to FIRST, or if they'll even bite. Couldn't hurt to tell them though. We'll see what happens.

Re: The communication tides are shifting...
 

I am a huge Vex supporter, we hosts two large events with multiple fields every year but there is no way you can compare Vex to FRC in this way.

In Vex your controllers are literally just a few feet away away from the robots. They don't have the high powered motors and other noise generating devices and you don't have 140 monsters running around at high speed that can cause serious harm.

Also just see how well it works when you have a couple kids with smart cell phones in their pockets when running a match.

Re: The communication tides are shifting...
 

Speaking of Vex (specifically the Cortex and VEXnet control system),

-I recently FTA'd a small qualifier (24 teams) in my high school gym. I used three 1.4ghz Celeron laptops donated to me by the school when they got new ones and were literally throwing the old ones that still worked away. One ran the field and audience projector, one ran the skills challenge field, and one ran the two pit displays (rankings on the projector, SC rankings on the laptop screen). We never had any field issues with the scoring computer at all, and ran only one test match. All issues were solved by plugging in the cables or replacing the VEXnet keys (they seem to fail when they heat up a lot after an hour or so of running continuously, but it's not a permanent failure). We did not have a field delay EVER. The DJ (a FIRST student) told me he had a list of dancing songs, and couldn't play them because the field worked all the time. VEXnet is based on 802.11, has connection times under 20s, and allows radio use in the pits.
-The entire cost of the VEXnet field controls was sub-$200 for all of the field control electronics and cabling, plus the requirement for a WinXP laptop. You can download it from http://www.dwabtech.com/main/tm2/ and play with it to answer any of your questions.

-We also use the VEX Cortex control system in OCCRA, where the robots are similar to FIRST:
-115lbs including battery, extra 5lbs if you use pneumatics with the on-board compressor
-no bumpers
-28x38 footprint by 40" tall
-Motors include CIM motors, DeWalt drills, and various automotive window, wiper, van door, seat, etc. motors
-Our Cortex was right in the middle of our robot, in a fairly bad place for interference. This is due to an OCCRA rule requiring all electronics to be in a box for protection of the electronics, and the only place the box would fit was exactly in the center.

-At OCCRA, they had very few field issues all season, and the OCCRA field staff was very good about checking all robots communications before each match, and diagnosing all failures. In OCCRA, there are 8 VEXnet systems on the field at a time, plus any teams running VEXnet in the pits (which is allowed).

As some of you asked about the FRC FMS, here is what I can tell you (I know this is certain):
-The Scorpion Case has a PLC and the FMS server, both connected via Ethernet
-The Cisco AP, sitting on top of the Scorpion case
-The blue start/stop button box, connected via Ethernet
-The SCC's (Station Control Cabinet) sit at each alliance station, and contain:
--Allen-Bradley hardware including an industrial Ethernet managed switch
---This has ports for the Scorpion case, each robot DS, and the switch below
--A generic Ethernet switch - This is for all of the Allen-Bradley remote IO at this end of the field, such as any lights or e-stops or sensors
--Various Remote IO boards for all of the IO at each end of the field, all communicating over EtherNet/IP
-The ref tablets are Allen-Bradley touchscreens, also connected via Ethernet.
-Each driver station is VLAN tunneled to it's virtual WiFi network on the Cisco AP. The rest of the field runs VLAN's for its own use, I believe only one but I am not certain.

Edit: More information
-The lighting on the bridges is DMX controlled, via DMX-Ethernet bridge which is connected to the Ethernet switch on one of the SCC's
-The lighting on the driver stations is controlled via the Remote IO boards
-The bridge sensors are also connected to Remote IO boards
-All of the wiring to the bridges runs under the barriers/bumps. They act as conduits.
-The blue box just duplicates software buttons and the field can run without it

I won't attempt to explain any software, as I'm not an FTA and can't verify my knowledge.

Re: The communication tides are shifting...
 

Question:

- Are all robots Crios and Cameras operating on a single common Wifi Channel ? I believe so. When there are a number of non-overlapping wifi channels available, why would we not make use of them (bit more configuration hassle). Put the camera streams on one channel and data on another. Or 2 robots out of 6 on each channel. Yes the field would need 3 radios I believe.
- also, at a competition where 4 fields x 6 robots are all on the same channel and close enough to each other the RF collision and retry rate must be enormous.

re the comment about the CRIO's being bullet proof. I don't disagree but there is still (always) room for improvement. I have seen a number of glitches and failures where better diagnostics and reporting would greatly help: CRios rebooting randomly, cause unknown, chips blown off their PCBs in the I/O cards. A number of dead $150++ I/O cards.

Also it would be so productive if the CRIo could reboot and reconnect in a second or two. It is a realtime system.

Re: The communication tides are shifting...
 

I saw this earlier in the thread and I know Don touched on it but this is simply not true. Security though obscurity is never the answer.

We have literally hundreds of engineers who want nothing more than to see FIRST succeed for the students (and a lot of intelligent and informed students too). Letting them see how everything works and interacts can't be bad (although we know how much people on CD love to complain, that could be a problem :rolleyes:).

Re: The communication tides are shifting...
 

Ubiquiti Networks, for one, makes a 3.65GHz CPE unit that is in the $100-150 price point, intended for WiMAX use. These get installed at people's houses in remote areas for wireless internet. They have to get an FCC (US)/CRTC (Canada) license to transmit on those frequencies, but the fee is minimal (I think $50-100), and the license easy to get, because you're transmitting at < 1W.

If FIRST got the FIELDS licensed as 3.65GHz mobile transmitters (not unlike a TV News van), then the TEAMS wouldn't need to do anything special.

Lots of similar equipment exists in the mid 3GHz bands from a variety of manufacturers.

I actually had 2 Ubiquiti NanoStation2s some time ago, and was able to create a 2.4GHz 802.11g network bridging the two, some ~1.5km away, with a stronger signal than my laptop was picking up from a Linksys WRT-54G a few inches away.

I was ALSO able to use one Ubiquiti NanoStation to create a 2.4GHz 802.11g network I was able to pick up with my laptop's internal NIC from some ~2km away, given Line of Sight.

Ubiquiti's gear is built for outdoor use through all sorts of nasty weather, being mounted on masts hundreds of feet in the air. Its built strong enough for FRC's abusive environment.

Re: The communication tides are shifting...
 

I have one compare that is comparable - BEST. I have been a coach for BEST Robotics for the past 9 years, and BEST recently (last 2 years) moved to the VEX control system. I have been nothing but disappointed by the VEX system used in BEST. Our robots have had many "communication issues" stemming from random drop-outs to random reboots (all while using the default code), and at least 3 tournaments here in Texas had to go to using tethers because the wireless just could not be trusted - when teams play 8 rounds and have a combined connection time of less than a minute, there's something bad wrong.

I'm not here to blast the VEX system, I'm just here to say it's not the bastion of hope everyone is looking for.

-Danny

Re: The communication tides are shifting...
 

As a counter point. I have been running VEX events since 2008 Houston VEX.
When the CORTEX system was first introduced in the college division in 2009 there were definitely bugs and errors.

However in the past 2011-2012 year. I have seen a significant step in quality and reliability in the VEX CORTEX robot and field system. I have helped field tech four events in Texas (Galveston, Houston, Austin Fall, Austin Spring) combined together that's roughly around 400 matches.

Although I'm going from memory, I have seen less than 10 bad matches out of 400 matches. The errors from these matches are typically not using the standard programming template or failure to sync keys.

This coming season, I will carefully document each communication failure to keep better documentation of the field issues experienced with VEX. However I can attest that the system is extremely reliable from years of experience.

Re: The communication tides are shifting...
 

Let me put on my "day job" hat. :)
Speaking as the Director of Product Development for VEX Robotics, Inc...
We have been informed that certain BEST hubs had some issues, and are working closely with folks at BEST Robotics, Inc. to improve their customer experience, specifically through education of folks on how to troubleshoot the system.

Danny -- I'd like to think you and me "go way back." I'm not sure exactly what is going on, and I'm wondering why you haven't emailed me already? Have you been in touch with the VEX Support Group? As you should know, we'll work to "make it right." jvn@vexrobotics.com

That said... at VEX World Championship this year (an event with over 600 teams) we played 1,900 tournament matches and over 450+ skills challenge matches with almost no issues.

-John

Re: The communication tides are shifting...
 

Please explain this quote a bit more. Why does it need to be more complex for it to be robust and safe? Isn't the opposite usually true? Besides, I believe the VEX TMS is already more powerful than the FRC FMS, while also being more stable, easier to use, and less problematic. VEX Worlds ran 1900+ matches and 650+ skills matches without a single problem attributable to VEX TMS.

I still don't get this. You define a sensible language as one that an FTA can debug in real time? You honestly want FTAs to dig into labview code during an event? If the problem gets that far, the process to develop a robust FMS has failed big time. I define a sensible language as one that allows the developers to meet the usability & performance requirements of the event, nothing more.

Re: The communication tides are shifting...
 

Even ignoring the still-mysterious loss of control experienced by a number of robots, I think the multiple issues that have been traced to the D-Link wireless bridge contradict any description of the system as "very robust".

Re: The communication tides are shifting...
 

Failsafe systems are rarely simple in implementation. The field management system has to be able to shut down robots, no matter what else goes wrong. From my experience in FRC since 1999, the first purpose of the whole control system is to make sure the robots shut down when they are supposed to shut down.

I can't speak to the VEX world championships, not having been there. But I have seen communications problems at the competitions I have been to. Not as many as at FRC competitions. But I have seen robots not shut down when they were supposed to. And again, I am not bashing VEX. I love the VEX system, and have a bunch of kits in my classroom.

Re: The communication tides are shifting...
 

I think we may be throwing too much technology at the FMS and hence introducing to much opportunity for bugs to creep in. At the off season scrimmages in Oregon we decided to NOT USE AN FMS because of all the problems. Each team's laptop uses a 802.11n USB dongle (or built-in 5Ghz support) to communicate directly with their robot. The philosophy is if it works in your lab, it will work on the field. Our matches at Girl's Generation and BunnyBots go off without a hitch. If a robot isn't communicating it's the team's problem. There's no mysterious "FMS" that people can point fingers at. The wireless adapters in the 5Ghz band seem to have no problem working out what channel to use and many robots last season were streaming high quality video. You also don't have to set the key on your radios at the event.

In a FIRST competition environment we could do the same thing but have a USB plug for the field to communicate with the driver's stations. That would be how you would disable a robot, how you'd start the match, switch from hybrid to teleop, etc. That USB plug could look like a keyboard to the laptop so you'd have keystrokes for all those things.

While we know this works at a regional with a single field I don't know if it can work at the Championships. We always have the on-deck teams turn on and connect to their robot while waiting to play. That means you'd have 48 teams connecting at the same time in the dome. They won't all be driving so it might be fine.

Anyway, I'd vote for keeping it simple.