Team Fusion #364, Bayou Regional, FMS Woes

[linuxboy]

Originally Posted by RufflesRidge

They do. I have also seen an AirTight sensor on the scoring table, I was curious what it was so I did some googling: http://www.airtightnetworks.com/home...-security.html

It sounds like you were trying to pull a channel setting off of the radio. he client does not choose the channel in a WiFi setup, the AP does meaning that either all teams or every team in a single station would likely use the same channel. Given Andrew's comment of only 3 antennas per frequency it is likely that all 6 SSIDs are broadcast on the same channel meaning all teams are using the same channel.

There are a couple devices on the scoring table, the AP, is what broadcasts the networks, the Air Tight, is very good at scanning surrounding networks, and can even shut them down (FIRST hq must be consulted before that happens). I believe FIRST actually has an FCC license to operate those. The air tight may be used for other things, but thats what I've been told about it.

[GrimmReaper]

Yeah i'd like to comment on my understanding of the field WiFi setup and maybe this will help clear up a few concerns. I am not an expert on the FRC Field, but i have extensive networking knowledge, I have not been involved in the field setup, but i've done a lot of research on it and I do believe I have a good grasp on this, as such - if i am incorrect - please correct me ;-)

Think of it like this, if you walk into starbucks with your laptop and connect to their wifi, you don't choose the channel you are on - the starbucks router does. If you and 5 other buddies all walk into starbucks and connect - you're all on the same single router, on the same single channel all connecting to chiefdelphi at the same time with no issues. The field is essentially the same situation - your robot routers in bridge mode are equivalent to a laptop at starbucks - but they are really a middleman allowing you to connect both a camera and crio to that network - but they are essentially conneting the same way.

Specialties for the FRC Field
1) I believe in most cases the field WILL be configured for 5ghz, all of our routers are capable of 5ghz, the lack of 5ghz on classmates is irrelevant as they are physically connected for the matches.

2) I'm unsure if every team is given an individual WPA password via the kiosk or if they are all the same WPA password, however the kiosk does set the ip address of the router for sure. Separate passwords could allow them to restrict acces a little better, but it can still be done with a single shared key.

3) From what i've gathered - the field will only accept traffic from those teams that are scheduled on the field, so in theory any additional potential traffic from other teams in queue or elsewhere should not be present on the field network during a match.

4) all teams while connected to the field are essentially part of one big network, in theory you could all connect to each other, and you are all able to connect to the field FMS Server, kinect kiosks, etc... However this traffic COULD easily be limited on the networking side, but i'm unaware if the FRC Field network sets up anything to segment the team networks.

5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...

If i left any details out, let me know... but i believe that's a fairly good representation of the field network.

As for the actual problems that teams experienced and possible causes, i wish i could add some insight to that, i've been racking my brain on it for the past couple of days, watching tirelessly for someone to come up with the magical answer so i can stop wondering as well :-)

Good luck to everyone!
- Jon

[EricS-Team180]

Originally Posted by Mark McLeod

There's a body of anecdotal randomness...
Engineers measure things...

Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."

Do you have any suggestions of what we can measure Mark?
Eric

[RufflesRidge]

Originally Posted by GrimmReaper

5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...

There's an SSID for each team on the field set to the team number (e.g. "2046"). You can see this with a laptop with 5GHz capabilities, or by logging into your bridge and looking at the settings.

[linuxboy]

Originally Posted by GrimmReaper

Yeah i'd like to comment on my understanding of the field WiFi setup and maybe this will help clear up a few concerns. I am not an expert on the FRC Field, but i have extensive networking knowledge, I have not been involved in the field setup, but i've done a lot of research on it and I do believe I have a good grasp on this, as such - if i am incorrect - please correct me ;-)

Think of it like this, if you walk into starbucks with your laptop and connect to their wifi, you don't choose the channel you are on - the starbucks router does. If you and 5 other buddies all walk into starbucks and connect - you're all on the same single router, on the same single channel all connecting to chiefdelphi at the same time with no issues. The field is essentially the same situation - your robot routers in bridge mode are equivalent to a laptop at starbucks - but they are really a middleman allowing you to connect both a camera and crio to that network - but they are essentially conneting the same way.

Specialties for the FRC Field
1) I believe in most cases the field WILL be configured for 5ghz, all of our routers are capable of 5ghz, the lack of 5ghz on classmates is irrelevant as they are physically connected for the matches.

2) I'm unsure if every team is given an individual WPA password via the kiosk or if they are all the same WPA password, however the kiosk does set the ip address of the router for sure. Separate passwords could allow them to restrict acces a little better, but it can still be done with a single shared key.

3) From what i've gathered - the field will only accept traffic from those teams that are scheduled on the field, so in theory any additional potential traffic from other teams in queue or elsewhere should not be present on the field network during a match.

4) all teams while connected to the field are essentially part of one big network, in theory you could all connect to each other, and you are all able to connect to the field FMS Server, kinect kiosks, etc... However this traffic COULD easily be limited on the networking side, but i'm unaware if the FRC Field network sets up anything to segment the team networks.

5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...

If i left any details out, let me know... but i believe that's a fairly good representation of the field network.

As for the actual problems that teams experienced and possible causes, i wish i could add some insight to that, i've been racking my brain on it for the past couple of days, watching tirelessly for someone to come up with the magical answer so i can stop wondering as well :-)

Good luck to everyone!
- Jon

I don't mean to be mean, however some of that is incorrect I will go through the items and comment about my understanding.

1) I agree
2) Each team has a separate WPA key, they are generated Wednesday night I believe
3) This is done by having a separate SSIDs (unbroadcast I _think_, but the same as the team number I would guess) and keys for each team
4) Each team has it's own VLAN, I'm not too good at VLANs so I can't describe this too well, FMS can talk to anything, but each DS cannot talk to other DSs or other Robots (this is why, for those of you that have done field setup, each player station has a labeled port on the SCC, instead of just plugging into the switch)
5) The field AP is reconfigured every match, with the SSIDs for the teams in that match, with the appropriate WPA keys. This is why the scorekeepers must "prestart" the match. The SSIDs are not broadcast I _think_

What you described seems almost like it could be a FMS Light setup for a competition.
I know these things from doing quite a bit of volunteering, and asking the FTA a lot of questions while I do.

I agree, though with that last bit.

- Oliver

[RyanN]

Originally Posted by EricS-Team180

Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."

Do you have any suggestions of what we can measure Mark?
Eric

I have all of Team Combustion's logs from Bayou. I have close relations with them since one of their mentors was my mentor at the NASA Stennis Space Center for four internships. We're a sister team of theirs.

I'm focusing (read: trying to focus) on my Advanced Circuits exam I have tomorrow morning at 8AM. After the exam, I'll review their data, try to match up some data with ours and do some comparisons. I'm really interested in seeing what the data will show.

I have send the logs to Greg as well.

I'm still sticking with too much WiFi traffic and interference for our connection problem until I figure out otherwise.

[Gary Dillard]

Originally Posted by Mark McLeod

There's a body of anecdotal randomness...
Engineers measure things...

I'd have to disagree with you semantically here Mark. If team A loses comms or their cRIO resets during match 1 at a certain location, that's a data point. If they don't during match 2 at the same location, that's another data point. If team B loses comms at a different location, that's another data point..... specific events and frequency of occurence is data. Pass/fail tests are data as well - it always worked when we did this. The problem is, engineers tend to ignore things that can't be quantified (qualitative data). If a jet engine on the test stand starts spitting parts out the back, it doesn't really matter whether the sensors indicated something went wrong or not - the evidence speaks for itself.

The FMS' job is to communicate with the robots, and it isn't happening hundreds or thousands of times per season. Are the vast majority of the problems robot issues? Probably, but multiple teams have done extensive troubleshooting and don't have issues when they're not running through the FMS. If it's performing as designed, then it's designed wrong. If it has some limitations, either they are not being communicated well to the teams or no one is interested in looking for them.

[Greg McKaskle]

I think Mark's point is that measurements result in data, numbers, etc. They are independently verifiable and repeatable. Anecdotal data is valuable too, but you need to apply valid statistics to it and work hard to remove bias. Engineers act on hunches and anecdotal data all the time, but that doesn't make the data they acted on to be measurements.

If a team member walks to the FTA and points out that five of the field lights are not working, that is an easily verifiable, quantitative measurement. We all have sensors/eyes for making that measurement. We also have experience helping us to evaluate the impact -- are the clustered or spread, where are they pointing, do we have spares, which should we replace to make the most impact, etc.

If humans had RF receptors, comments about a dead spot would probably be acted upon similarly. But since we don't, I'd hope the FTA would scale the reports by a credibility factor and file it into the "something to watch for" bin. I suspect that if they also saw consistent secondary evidence of a dead spot, they would call Manchester and see how they can make a hard measurement, or verify their hunch, or switch to an alternate setup recommended by FIRST.

RF is complex and not intuitive to those not specifically trained in it -- including myself. Wifi builds on RF, and adds its own 30 years of protocol complexity. The robots have their complexity as well, and as I've said before, the majority of the time, the issues can be demonstrated to be a robot or DS issue.

In the best cases, lets call them category-one, we can make primary and repeatable measurements that everyone can learn from. We can measure the battery to determine if it was at fault. We can jiggle wires and show a particular wire that causes the issue -- because I can reproducibly cause the fault and we can all see it. We could also scope it if necessary and measure the circuit fault and give numerical data to the circuit fault, but that usually isn't necessary. Problems like these result in good learning opportunities.

Other times, category-two, we can only eliminate and verify that common issues aren't present. We replace components until the lack of a fault gives us a likelihood that we have identified the issue. These are far less satisfying and prone to misinterpretation. Often we replace components and the issues doesn't go away. Does that mean the component that was removed was good, or simply that it wasn't the only bad component. In a system where multiple elements can fail and the symptoms look similar, this can be frustrating, and it is hard to make it into a good learning opportunity. Whenever possible, direct and independent measurements of the simplified system will help turn this into the former case where we are all learning more and more satisfied with the outcome.

I believe 364 did a very good job trying to troubleshoot their situation. I wasn't at the event, but I believe that both they AND the FTA and FIRST staff were drawing conclusions based on an educated interpretation of sketchy data. I am trying to be objective and glean what I can from the data. I am reviewing their logs, the logs of other teams at the event, and making my own logs of known failure points to try and identify a set of likely causes and eliminate others. If nothing else, I would like to put the right tools in place so that more of the category-two issues turn into category-one issues. If other teams have logs they would like to send me, PM and I'll give you an email destination.

Greg McKaskle

[Dusk Star]

Originally Posted by linuxboy

3) This is done by having a separate SSIDs (unbroadcast I _think_, but the same as the team number I would guess) and keys for each team
- Oliver

I can confirm that the SSIDs are broadcast, and that they are named with the team number; the first competition I went to this year with my laptop I looked at the wifi networks and immediately went "someone is going to get yelled at for connecting over wifi..." After a match or two, I saw that the SSIDs were those of the ones currently in a match, and started to ignore it. I don't know whether they were 5ghz though, as I wasn't about to open a wifi diagnostics tool just to see what network they were on. I do know I have both frequency sets, though- 3x3 wireless a/b/g/n card

[GrimmReaper]

Originally Posted by linuxboy

I don't mean to be mean, however some of that is incorrect

You're not being mean, I asked you to correct me if i was wrong :-)

I'm just happy to have more detailed information about the FMS specifics, thank you for that. It's actually architected a bit better than I expected which is good to hear ;-) Thanks to everyone that helped clear up my incorrect details.

So just to clarify my points from earlier on the channels - we still have no control over channels, that is going to be assigned by the FMS router, however if we are all on different SSID's they could be all the same channel or all on different channels. If they're on different channels, how do they allocate the channels? at random during pre-start? are they pre-set for each station i.e. Red-1 is always 5ghz channel 11, or something like that? I still don't see how that could cause any of the issues 364 saw, unless they were always on the same channel and it was a noisy channel... It would be good to know how they allocate channels.

- Jon

[Natchez]

Ryan,

Thank you very much for the kind acknowledgement BUT, for the record, I was only one of very, very many that helped get the ball rolling for Fusion to have an opportunity to compete at Lone Star. With that said, it is a pleasure helping a team with such a rich history of helping others.

----------------------------------------

Everyone,

For the most part, this has been a good discussion along with good input BUT we still don't have a good answer for what has happened much less a root cause for 364's problems. The Lone Star Regional is very interested in helping 364 and FIRST determine the root cause of these problems. In this spirit, please keep posting ideas of what we should do at the Lone Star Regional to help troubleshoot this problem; we will develop a troubleshooting flow based on everyone's input.

Unless we get a better idea, our first test will be to unbag 364's robot on Thursday and attempt to hook it to the field (e.g. take robot out of bag, put battery in robot, get WPA key, etc.). We also intend to get other robots that have successfully connected at other regionals (57, 118, 231, 418, 624, 653, 1429, 1477, 2415, 3335, etc.) to test their field connections and so we can also have 6 robots connected at the same time. Depending on these test results, we probably have hundreds of options for what the next test should be; if you have any ideas, we'd like to hear about them and we don't care how silly they are (e.g. if 364's robot has no problems connecting, ask everyone in the arena to start turning on the wireless on their laptops and call their mothers from their cell phones to see if we can get 364's robot to shut down ..... if 364's robot will not connect, reconfigure 118's robot to look like 364's robot, use 364's bridge and see if 118's robot connects - the regional may even supply a "Basic Bot" that we can reconfigure). I will guarantee you one thing, our FTA will take everyone's input and will have a rock-solid troubleshooting flow when he walks in the door on Holy Thursday.

Thanks & give us your troubleshooting ideas,
Lucien

[Tom Bottiglieri]

Originally Posted by Natchez

Thanks & give us your troubleshooting ideas,
Lucien

Knowing what part of the system is broken quickly should help.

I would run tcpdump at three different places and compare the captures to find where packets are getting held up or lost. Run one on the DS (or on the same switch), one over the air (using the team's SSID and PSK), and one behind the robot bridge. You can probably use one laptop to do both of the latter.

Basically what you are looking to do is trace packet flow. Check the timestamps of packets as they move from the DS to the AP to the bridge to the bot. It will be pretty apparent if there is a faulty piece of hardware in the chain. (I'm not as worried about the Cisco AP as I am the Dlink bridge you can buy at Walmart...)

If packets aren't getting delayed or dropped anywhere in that chain, check to see how it takes for replies to be generated by both the DS and cRIO.

PS. You can use Wireshark instead of tcpdump for prettier output, or if you want a GUI.

[Dale]

This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.

At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.

I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.

The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).

[Jimmy the Kidd]

Originally Posted by Dale

This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.

At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.

I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.

The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).

This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?

[dez250]

Is there an update on how 364 faired at connecting and working during practice day at Lone Star?