Re: The communication tides are shifting...
 

I do have to say a few small points, for what little it is worth. I worked as a Robot Inspector at both Virginia and North Carolina, and at the latter we had one particular culprit, that while it was a brilliant system, the implimentation to begin with was causing lag.

We were looking at the FMS data in realtime and interfacing with the teams to help solve problems (to my knowledge every issue we had was either related to robot code, low batteries due to the power shutdown during the night, and a few lose connections... no FMS issues). The one team that was initially causing lag had three cameras streaming at an extremely high refresh rate. They toned it down to an acceptable rate as soon as they were informed it was causing lag not only with their bot, but with other robots on the field.

These bandwidth issues seemed to cause lag, not any kind of complete and total interference with a single robot (ie. Red1). Feel free to correct me at any point in time, but those were my observations from behind the Pilot's seat.

Re: The communication tides are shifting...
 

I have a contact at Quantenna Communications... For my co-op, I did wifi testing with their demo boards. It took quite a bit of distance to stress their product (on the order of 300-400 feet in a commercial building). I'll tell them they might have an opportunity for this niche...

Re: The communication tides are shifting...
 

The thing that keeps getting me is the fact that in all of the regionals I have been to, all of the feeds I have seen of champs, all the issues that come forward in the latter parts of the competition are robot issues... our troubleshooting at NC proved that! However, early in the game at any of them there are things that needed working out with the FMS... minor quirks.

We are talking about top notch teams here, NASA supported and more! Teams that won nearly every single match, teams that know what they are doing, enough to make it to Einstien in the first place!

In the hundreds, hundreds of matches that FIRST and this FMS system and the D-Links have gone through for all of FIRST... why was this problem not more common? I am nearly positive from the facts before me (what few they may be) that the D-Link is not to blame here. Something on Einstien's hardware had to be off or not optimized.

The only other thing that occurred to me is the correct setup of the routers. They went from one field to another, the same as going from our workshops to competition, or the practice field. If the routers were not configured correctly we would have the same issue.

What it is, I do not know. I am happy that the D-Link/FMS system works as reliably as I have seen it work in the past year... I saw very few issues at VA and NC... but I am severly dissapointed to hear that the Einstien hardware was never even tested (though I would love to be told otherwise).

Sincerely,
Petrie

Re: The communication tides are shifting...
 

My team's robot (3142) has never had connection problems. A lot of you are saying the control system dropped your team's connection multiple times, but did anyone here do FTC last year with Samantha? Rankings in FTC were largely determined by how many times your robot worked, because no one's robot worked more than 80% of the time. You don't realize how well the FRC FCS works.

I think what FRC has to do with the control system next year is give themselves a large amount of headroom on the bandwidth, and allow use of a more expensive, more reliable radio, along with an inexpensive option for all these new Boys and Girls Club teams. FRC could also take a radio and put an alternate firmware on it (dd-wrt), use QoS for FCS messages, and throttle bandwidth on accessory streams. This could also make flashing the radio quicker, since FIRST controls the protocol.

Also, a lot of control problems could be team-related, as impossible as that seems. Going back to FTC (experiencing a similar problem), my team had constant connection issues last year. This year, my team got a new Samantha, a new brick, new code, a new RobotC version, and there was a new FCS. The only shared factor between the two seasons was me, the programmer. My team still had connection issues. It has to be me, somehow, because most other teams had much fewer connection problems. Multiple people looked at my code, and couldn't see a problem. Yes, FTC is not FRC, but parallels can be drawn.

Re: The communication tides are shifting...
 

The point with the cRIO and WiFi connection is that we can get experience with real-world technologies and practices. The IFI controller and its radio were unique to FIRST, so they didn't help us in that way.

If the D-Link or TCP/IP were to be scrapped for something better, what would work well that other people/industries actually use?

I keep thinking of the DIY hobbiest community and its similarities. What would people use if they were making a robot? Arduino ethernet and wifi shields come to mind, with XBee modules. RC cars, airplanes, and helicopters also come to mind, what technology do they use? (I mean pro hobbiests, not $20 RadioShack toys).

And if these are not applicable or too hobbiest oriented, what technology do they use in industry? How do iRobot's military robots communicate? Boston Dynamic's Big Dog? All those quadrotors in development by universities? The ones companies can buy?

Re: The communication tides are shifting...
 

Hobby wise, at least for remote control, I believe Spektrum is the go to for radios/receivers. They use the 2.4GHz ISM-band and some fancy techniques and redundancies for improving reliability. People fly heavy, multi-thousand dollar R/C aircraft with these things so they certainly don't want any loss of communication. But again, this is an apples to oranges comparison as Spektrum controls the protocol from top to bottom.

Re: The communication tides are shifting...
 

I was under the impression, through the rumor mill, that FIRST had intended to deploy OpenWRT on future WiFi hardware. I know from my own experiences building my own robot control system with Parallax Propellers, a NetGear router and DD-WRT that there is more control to be gotten there.

I think it's too soon to assume we need to scrap WiFi hardware. It's probably not to soon to assume we need to scrap software for WiFi routers designed to work in them while they are stationary.

A hacked firmware would open the door to in-access point diagnostics.

Re: The communication tides are shifting...
 

Spektrum, Futaba, JR, and I think there's another brand all use the same 2.4GHz frequency. Each has its own set of hardware, but any Spektrum reciever can bind to any Spektrum transmitter (within a certain amount), and the same for the others. You could have huge flights with everyone using a different setup, and not a plane will go down due to interference. Even with all the transmissions from all those transmitters.

If we didn't need all these sensors and the FMS, I'd say to try using R/C controllers...

Re: The communication tides are shifting...
 

If we just had people from each of the robot communication companies, it would probably help a lot since they would probably know more about their product than we do.

Re: The communication tides are shifting...
 

I think the FMS hardware that resides inside the Scorpion Case has been tested along with the PanelView screens that all of the ref's access before the rig was shipped field components and FMS gear. The field components were brand new but the FMS gear is also from last year. As far as I am aware the concept behind the AP controller getting controls from FMS about match information in the pre-start period and then passing it to the AP has remained the same ever since switching to the bridge setup.

This issue has happened on field all across the nation last year and this year. There are 18 trucks plus the one that resides on FedEx HQ and probably all of them have experienced that field issue.

The thing to do now is to go through all of the field data and FTA logs about which field did what under certain circumstances in certain environments. And I am sure FRC Engineering department will be busy all summer preparing a new fix for next year.

Re: The communication tides are shifting...
 

I think this task may not work for some cases. I can easily see with all the possible issues that some cases are in fact robot issues.

Team 11 had such issues and when the field logs were reviewed all that was discovered was that the robot lost communications.

The troubleshooting process entire simply needs to be expanded as well as the monitoring. If the logs were in fact highly detailed I think that the diligent field personel would have spotted faster ways to put a stop to what happened during Einstein.

In any event I agree this problem must, can and will be solved.

Re: The communication tides are shifting...
 

I think we may be missing the obvious solution.



</troll>

Re: The communication tides are shifting...
 

Begin long and rambling explanation:

I had the chance to talk with an engineer from NI at great length about what goes on inside the field system. Over the course of a few hours, I am fairly sure I asked about every single aspect of the field and how it works (and more importantly, why it doesn't). This is what I was told:

The field itself has two PLC's that handle the scoring hardware and a PLC that handles the bridges. These PLC's talk serial over IP using bridges to connect to a dedicated LAN. These PLC's are only restarted maybe once a regional, and are very resilient. The only time they have to be restarted, is if for some reason the system experiences a power surge that causes one to 'disappear' while it restarts. Next in line on the field are the PanelView systems, which are just win7 tablets with touchscreens running a simple interface to talk with the scoring computer, which is itself a part of the FMS, but that in a minute.

Next in the field is the Robot, the robot has a relatively new cRIO real time controller that is very advanced compared to what FIRST had previously. This system is VERY robust, considering that student programmers are writing code that is for the most part poorly thought out. There are many teams that write good code, but there are many more that don't. The only point at which the code on the machine can easily be blamed with 100% confidence is CAN. The CAN bus system was never designed to work like this, where the machine is expected to do a rapid start up and perform dynamic hand-offs of the CAN master node (the switch from auto to tele). Assuming that CAN is not in use, and that there is good code on the machine, the cRIO is a bulletproof system.

On the other end of the line is of course the DS, the DS is a simple labview program that communicates with the robot to convey the info from the hardware connected to the DS. In my 3.5 years of FRC, I have never seen a problem with the DS. The Dashboard on the other hand, is full of holes. When teams use the smart dashboard, they are asking for trouble, the same goes for all the other dynamic dashboards. These programs are too broad in scope and attempt to fit too wide an application. The only way to write a truly stable dashboard is to write it yourself for your own robot.

So now we are at the physical connections in the system. The IP network is run using Cisco managed hardware that is, for what it's worth, very good equipment. That being said, there are a few bugs. For one, the main WiFi transmitter is just one unit. It is a very special, very powerful unit, but it is still just one transmitter. There is also the issue of it just being Cisco equipment. For that one, some background is needed: when a router is approaching the maximum amount it can handle, it will start dropping packets, TCP which allows for this, will simply request the missing packets at a later time, when there will presumably be less network usage. But here is where the fun part is, how does the router decide where to drop data from? Easy, starting at the lowest numbered numerical address, moving up. This easily explains why old teams (low number) see many more connection problems than the rookies (high number). This is unfortunately just the way the cookie crumbles, so there is no way around it short of coming up with a gigabit WiFi system.

So now for the last part, the FMS. The FMS is a fully redundant system consisting of two rack-mounted servers in the scorpion case. This system is the week link of the entire field. The FMS itself has to handle ALL of the following hardware/software functions:

• 9 VLANS
• 2 PLC's
• 3 Monitors
• 18 dynamic data sources
• A twitter feed (not much here, but it all adds up)
• The sundial system
• WPA encrytion cycling
• Match Scheduling

To top it all off, the whole thing is written in VB.NET! If you want a laugh, head over to AndyMark, where under the field rental section there is a place to download FMS light, which is the same program, just without the bits that handle the Field hardware. You will quickly find that it is only barely stable, and that the it suffers from huge amounts of odd quirks inherent to VB.NET apps. It is clear to me that the FMS program itself is the root of this evil. Since it handles the WiFi system, it is also to blame for the oddness of connections that work perfectly on one field but not on another, or robots that work on the practice field, but not on the real field. The FMS program is in dire need of a re-write, preferably in LabVIEW, which most FTA's can debug errors from very quickly.

Some of you may remember from watching the Alamo regional Finals that there is a very awkward pause during the finals right after one team called for their backup robot. Look it up on Blue Alliance if you haven't seen it, it is one of the better times the judges have had to stall. When I asked later what happened, the Score keeper (who actually sits at the FMS console the majority of the time) said that the system just refused to take the team substitution. It wound up requiring the entire field to be reset and rebooted. This was an issue that had never come up before, but was attributed to a rare access violation within the FMS program.

All in all, we are dealing with a very robust system. The only part of it that is not robust is the FMS program itself. To me, the only solution is to scrap the current VB.NET program entirely, and rewrite it from the ground up in a more sensible language like LabVIEW.

Interestingly enough, we still have our Maize Craze robot in (what we think) is workable condition. We might bring it to the championship pit next year for kicks and giggles.

Re: The communication tides are shifting...
 

Firstly, let me thank you for sharing all that information.

Secondly:
No language insures in the absolute sense you can't have an error at all.

I share some of your lack of enthusiasm personally for .NET languages, however that doesn't mean that I can ignore that a process that produced the buggy product in the first place is prone to be just as buggy on the rebound (pun intended).

If it turns out the that FMS system is quirky they should troubleshoot, repair, replace as necessary.