Team Fusion #364, Bayou Regional, FMS Woes

[EricS-Team180]

After mchristina444's post, it reminded me of what George had told me. So I just wanted to throw it out there so others more knowledgeable could add credence or call shenanigans. So thanks to apalrd for that Yet, let me ask a different way: Is it possible for there to be wave cancellations in the near field? ...some odd RF effect? Just curious.


I'm with Gary, though. We're not here to pick on anyone, but there is a body of evidence collecting here. I know people like Greg M. are following up and I am grateful for that. Let's see where the data leads.

Eric

[Mark McLeod]

Originally Posted by EricS-Team180

... there is a body of evidence collecting here.

There's a body of anecdotal randomness...
Engineers measure things...

[wireties]

Originally Posted by Mark McLeod

There's a body of anecdotal randomness...
Engineers measure things...

So true - I hate what happened to the teams in this thread but there is no evidence (yet). Let's say the FMS said "OK, this is our fault". What direct action (not redesign the system) could they take to fix it? We can't tell them what to fix!

[EricS-Team180]

Originally Posted by Mark McLeod

There's a body of anecdotal randomness...
Engineers measure things...

Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."

Do you have any suggestions of what we can measure Mark?
Eric

[RyanN]

Originally Posted by EricS-Team180

Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."

Do you have any suggestions of what we can measure Mark?
Eric

I have all of Team Combustion's logs from Bayou. I have close relations with them since one of their mentors was my mentor at the NASA Stennis Space Center for four internships. We're a sister team of theirs.

I'm focusing (read: trying to focus) on my Advanced Circuits exam I have tomorrow morning at 8AM. After the exam, I'll review their data, try to match up some data with ours and do some comparisons. I'm really interested in seeing what the data will show.

I have send the logs to Greg as well.

I'm still sticking with too much WiFi traffic and interference for our connection problem until I figure out otherwise.

[Natchez]

Ryan,

Thank you very much for the kind acknowledgement BUT, for the record, I was only one of very, very many that helped get the ball rolling for Fusion to have an opportunity to compete at Lone Star. With that said, it is a pleasure helping a team with such a rich history of helping others.

----------------------------------------

Everyone,

For the most part, this has been a good discussion along with good input BUT we still don't have a good answer for what has happened much less a root cause for 364's problems. The Lone Star Regional is very interested in helping 364 and FIRST determine the root cause of these problems. In this spirit, please keep posting ideas of what we should do at the Lone Star Regional to help troubleshoot this problem; we will develop a troubleshooting flow based on everyone's input.

Unless we get a better idea, our first test will be to unbag 364's robot on Thursday and attempt to hook it to the field (e.g. take robot out of bag, put battery in robot, get WPA key, etc.). We also intend to get other robots that have successfully connected at other regionals (57, 118, 231, 418, 624, 653, 1429, 1477, 2415, 3335, etc.) to test their field connections and so we can also have 6 robots connected at the same time. Depending on these test results, we probably have hundreds of options for what the next test should be; if you have any ideas, we'd like to hear about them and we don't care how silly they are (e.g. if 364's robot has no problems connecting, ask everyone in the arena to start turning on the wireless on their laptops and call their mothers from their cell phones to see if we can get 364's robot to shut down ..... if 364's robot will not connect, reconfigure 118's robot to look like 364's robot, use 364's bridge and see if 118's robot connects - the regional may even supply a "Basic Bot" that we can reconfigure). I will guarantee you one thing, our FTA will take everyone's input and will have a rock-solid troubleshooting flow when he walks in the door on Holy Thursday.

Thanks & give us your troubleshooting ideas,
Lucien

[Tom Bottiglieri]

Originally Posted by Natchez

Thanks & give us your troubleshooting ideas,
Lucien

Knowing what part of the system is broken quickly should help.

I would run tcpdump at three different places and compare the captures to find where packets are getting held up or lost. Run one on the DS (or on the same switch), one over the air (using the team's SSID and PSK), and one behind the robot bridge. You can probably use one laptop to do both of the latter.

Basically what you are looking to do is trace packet flow. Check the timestamps of packets as they move from the DS to the AP to the bridge to the bot. It will be pretty apparent if there is a faulty piece of hardware in the chain. (I'm not as worried about the Cisco AP as I am the Dlink bridge you can buy at Walmart...)

If packets aren't getting delayed or dropped anywhere in that chain, check to see how it takes for replies to be generated by both the DS and cRIO.

PS. You can use Wireshark instead of tcpdump for prettier output, or if you want a GUI.

[Dale]

This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.

At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.

I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.

The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).

[Jimmy the Kidd]

Originally Posted by Dale

This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.

At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.

I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.

The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).

This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?

[dez250]

Is there an update on how 364 faired at connecting and working during practice day at Lone Star?

[James Tonthat]

Originally Posted by dez250

Is there an update on how 364 faired at connecting and working during practice day at Lone Star?

Lone Star is week 6.

[dez250]

Ah silly me, for a reason I thought it was this week. My mistake, time to wait out another week.

[bduddy]

As a mere field reset volunteer I don't have many details, but St. Louis was having a ton of connectivity problems on Thursday... far more than I remember from the last couple years.

[ChrisH]

Originally Posted by Jimmy the Kidd

This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?

It isn't wrong to hypothesize, but don't take the hypothsis as the solution until you have checked the data to ensure all data is accounted for.

When 330 was experiencing the same symptoms in Los Angeles they continued even after almost everybody but the floor sweepers were out of the building.

To reiterate, the cRio rebooted itself at apparently random intervals while running default code with all circuit breakers removed from the Power Distribution board, the camera was also disconnected. No extraneous APs were observed or detected.

[Dale]

330's problem seems different than what we were seeing in Oregon. There radios were just disconnecting.