Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan,

Thank you very much for the kind acknowledgement BUT, for the record, I was only one of very, very many that helped get the ball rolling for Fusion to have an opportunity to compete at Lone Star. With that said, it is a pleasure helping a team with such a rich history of helping others.

----------------------------------------

Everyone,

For the most part, this has been a good discussion along with good input BUT we still don't have a good answer for what has happened much less a root cause for 364's problems. The Lone Star Regional is very interested in helping 364 and FIRST determine the root cause of these problems. In this spirit, please keep posting ideas of what we should do at the Lone Star Regional to help troubleshoot this problem; we will develop a troubleshooting flow based on everyone's input.

Unless we get a better idea, our first test will be to unbag 364's robot on Thursday and attempt to hook it to the field (e.g. take robot out of bag, put battery in robot, get WPA key, etc.). We also intend to get other robots that have successfully connected at other regionals (57, 118, 231, 418, 624, 653, 1429, 1477, 2415, 3335, etc.) to test their field connections and so we can also have 6 robots connected at the same time. Depending on these test results, we probably have hundreds of options for what the next test should be; if you have any ideas, we'd like to hear about them and we don't care how silly they are (e.g. if 364's robot has no problems connecting, ask everyone in the arena to start turning on the wireless on their laptops and call their mothers from their cell phones to see if we can get 364's robot to shut down ..... if 364's robot will not connect, reconfigure 118's robot to look like 364's robot, use 364's bridge and see if 118's robot connects - the regional may even supply a "Basic Bot" that we can reconfigure). I will guarantee you one thing, our FTA will take everyone's input and will have a rock-solid troubleshooting flow when he walks in the door on Holy Thursday.

Thanks & give us your troubleshooting ideas,
Lucien

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Knowing what part of the system is broken quickly should help.

I would run tcpdump at three different places and compare the captures to find where packets are getting held up or lost. Run one on the DS (or on the same switch), one over the air (using the team's SSID and PSK), and one behind the robot bridge. You can probably use one laptop to do both of the latter.

Basically what you are looking to do is trace packet flow. Check the timestamps of packets as they move from the DS to the AP to the bridge to the bot. It will be pretty apparent if there is a faulty piece of hardware in the chain. (I'm not as worried about the Cisco AP as I am the Dlink bridge you can buy at Walmart...)

If packets aren't getting delayed or dropped anywhere in that chain, check to see how it takes for replies to be generated by both the DS and cRIO.

PS. You can use Wireshark instead of tcpdump for prettier output, or if you want a GUI.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.

At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.

I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.

The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Is there an update on how 364 faired at connecting and working during practice day at Lone Star?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

It isn't wrong to hypothesize, but don't take the hypothsis as the solution until you have checked the data to ensure all data is accounted for.

When 330 was experiencing the same symptoms in Los Angeles they continued even after almost everybody but the floor sweepers were out of the building.

To reiterate, the cRio rebooted itself at apparently random intervals while running default code with all circuit breakers removed from the Power Distribution board, the camera was also disconnected. No extraneous APs were observed or detected.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

330's problem seems different than what we were seeing in Oregon. There radios were just disconnecting.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Lone Star is week 6.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Try taking wireshark captures if possible with the field setup during the debug process for some post processing and clear evidence of the issue, if it's network related. http://www.wireshark.org/ and see this: http://wiki.wireshark.org/CaptureSetup/WLAN I haven't messed with wireshark in a while but I had to use it once for checking against some port scanning attacks on a network once, it's a pretty cool tool.

I would also try throwing on Tomato Firmware to some spare routers and set them up the same as they are in a normal match to see if you can get some more network analysis or if the firmware seems to fix the issue. http://en.wikibooks.org/wiki/Tomato_Firmware

Using either one of these utilities should make the issue stand out out like a sore thumb especially if it will be easily reproducible in Lonestar.

The last possibly could just be Ryan, I've never seen someone break so many macbook pro's in my life without touching any hardware but my boy RyanN can. I'm really just kidding on that aspect of course but seriously give some of these utilities a try.

-Brantley

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ah silly me, for a reason I thought it was this week. My mistake, time to wait out another week.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

As a mere field reset volunteer I don't have many details, but St. Louis was having a ton of connectivity problems on Thursday... far more than I remember from the last couple years.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

As a few people mentioned before, one of your tests should include swapping out the PD Board.

Kudos to Lucien and all of the others who made this trip to Houston available for team 364.

Andy B.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Does anyone know where the field in St. Louis this week was last week?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Changing the PD board is at the top of our list. It is the only piece of hardware that we did not replace at Bayou.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

The only reason we didn't change it was because it is a pain to change, and it would have taken too much time.

If the problems continue at Lone Start, then it will be replaced.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Everyone,

Thank you very much for your ideas & input. We will be documenting all of our testing that we conduct at the Lone Star Regional and will share it with you here.

Let the Problem Solving begin,
Lucien

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I think everyone wishes you the best of luck, also especially after what happened at the CT regional.

Best of luck 364 and all others who had FMS issues.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Test 1 completed: 364 successfully connected to the field and ran a match by themselves; similar to what 364 did Friday evening at Bayou with similar results.
... next test is to get other robots that send lots of data over the wireless connected with 364.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

what is the latest with fusion?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I just talked to RyanN on the phone. He said the robot is running great and they aren't seeing issues with the Lone Star field. Does anyone know where the Bayou field went?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Chuck Kennedy at FIRST would know that

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Everyone,

The executive summary with Fusion is, "All is well with 364 ... and we hope it stays this way." Fortunately and unfortunately, we found nothing wrong with 364's robot during our test runs. Here is the summary of test runs. A "(C)" denotes that the team was receiving camera video at the player station.

Test 1: 364 runs match by themself
Result 1: Normal operations

Test 2: R1-1429(C); R2-bypassed; R3-bypassed; B1-2936(C); B2-364(C); B3-118 (C)
Result 2: Normal operations

Test 3: R1-118(C); R2-1429(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 3: Normal operations

Test 4: R1-bypass; R2-118(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 4: 118 dies after being enabled for hybrid mode - Watchdog not fed - similar to what was seen in Connecticut; 364 & 3847 ran without issues

Test 5 (Same as Test 4): R1-bypass; R2-118(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 5: Normal operations

Test 6: R1-bypassed; R2-bypassed; R3-3103 (C); B1-364(C); B2-118(2xC); B3-418(C)
Result 6: Normal operations

Test 7: R1-2882(C); R2-364(C); R3-bypassed; B1-118(2xC); B2-647(C); B3-2585(C)
Result 7: Normal operations

At this point, we concluded the tests and declared 364's robot good-to-go. They have run a handful of times since the tests without issue; I'll let 364 update everyone with the specifics. The FTA, Jonathan, and his crew were awesome during all of these tests.

Finally, the engineer in me wanted to find a smoking gun with 364's robot but the other part of me is happy to hope that all will be well .... after all, it is the Easter season.

We'll keep everyone updated as the Lone Star Regional progresses,
Lucien

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

All is well with us.

We changed one thing. The router. During Bayou, we had tons of problems (obviously), so we used the pit admin spare. Firmware version unknown.

PD Board is still the same. We ran perfectly. I looked through some log files and saw no problems. I'll go through more tonight and report back.

Friday will be the day we would disconnect. Stands were basically empty. No interference.

The difference in the log files I looked are night and day difference from Bayou. We peaked out at 50ms latency at our highest point. Very little packet loss.

We're not holding our breath.

Thanks everyone that has helpe us out so far. We're crossing our fingers. We're here and ready to compete.

I'll keep everyone updated the best I can. I'm stuck with my mobile phone for the weekend.

-Ryan Nazaretian

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Here's hoping you kick some butt and have a solid regional outting. Good luck.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ya'll should have seen some of the smiles on the team's faces when the robot was first being tested. It was pretty special to see.

All the best, Fusion.

Jane

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I happened to be in Houston this week working, finished up today. So i plan on stopping by Lone Star to check things out Friday :-)

Good Luck!

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

So I browses through the log files and everything looks great.

Trip Time is 5ms on average. We have a consistent spike to 15ms every 30s or so. Non-issue. Assuming it to be a Windows service.

Weird phenomenon though. CPU usage continually increases. Usage starts around 40% and will raise to 45% throughout the match. It's non issue for a 2:30 match, but when practicing for 20 minutes, we can increase to 70% or so.

I wish I could post pictures, but that's a limit to the iPhone. :p

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

To Ryan and all of 364: GOOD JOB. I hope its stays good all weekend.
We (Mum and I) were thinking about you guys all day.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I know. Won't me like 3 Facebook messages. I told her I have text messaging. :)

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I was told it was from Bayou.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

This is interesting. What language are you using? If Java, I wonder if this is the garbage collector running. If C++, I don't have a good explanation (probably something in your teams' code). If Labview, I have no clue but suspect it might be memory management related (no idea how Labview actually does memory management under the hood).

EDIT: I assume you were referring to the cRIO's CPU usage. Is this correct?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

It's LabVIEW, and CPU usage is on cRIO. I could understand if I was doing something with arrays and processing them, but I'm not.

We're about to have our first match. Fingers crossed. Everything on the field looks good so far.

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Here is a play by play of Fusion in their first match:
Hooked to field
2 shots made in autonomy
alliance bridge to get balls
1 for 3
0 for 2
Coop with 624
...... 364 was fully operational

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

I watched the webcast to see you guys work!! I'm very happy that you're working, the engineer in me is a little sad because I want to know what the root cause of the problem was, but congrats on the co-op balance, and good luck in your future matches.

- Oliver

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

2nd match .... fully operational

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

To team 364: your robot is a very impressive and accurate shooter! I'm cheering for your team when I'm watching the webcast!:cool:

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

No issues all day. I'm too saddened we could not find the root cause. I'm also extremely curious.

Thanks again for all the help everyone has provided. I also want to thank all the volunteers at Lone Star that have helped us make it happen.

-Ryan Nazaretian

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Well, it is either the Bayou field/wireless environment, or your old radio, right?

If there is time after the event or during lunch tomorrow, maybe you can try the old radio?

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

great day for fusion, mazel tov and enjoy lone star. still i too would like to really know what was the issue at bayou.......

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan

We had a similar problem at the Madera tournament last week. We had a single good match and then the robot started going off line. Out of the remaining 10 matches, we had a single match where we had full connectivity between our robot, driver's station and the field. Most of the matches, we never were able to communicate with the robot.

Similar to what you did, we completely tore apart the robot throughout the competition. We started by replacing the Dlink, but eventually replace the software, the power distribution board, the digital sidecar and the CRio. We actually went through four separate routers and three power distribution boards.

The robot started to come back at the end of the competition and we maintained full connectivity at the final match. The problem was, by the time we go to that match, we had rewired so much and some of the connections were done incorrectly that the robot failed to perform properly.

In hindsight, I'm pretty certain that the issue was with the CRio because once we replaced that, the robot started to recover. We replaced that as the very last item. I definitely wouldn't recommend swapping out as many electrical components as what we did unless it's an absolute necessity. I don't think software was ever an actual issue with the robot not performing correctly.

This was extremely difficult to try and lead the team through. At the same time, I'm immensely humbled and grateful to the mentors and students from other teams, members from the FIRST competition, National Instruments, CSAs and FTAs that tried to help us get the robot back on line. I think that's the great testimony to the people involved with the FIRST program that they provided this much assistance.

We'll keep working our issue and try to find the actual failure. We'll also have to spend some time analyzing our team processes that might have possibly caused that failure.

Scott

Re: Team Fusion #364, Bayou Regional, FMS Woes
 

Ryan,
We just didn't have similar problems with what you experienced we had almost the exact problems you guys had. Even to the running all alone on an empty field Friday evening with no problems. Then coming back Saturday morning with the same problem. We had run only two practice matches on Thursday. The first with than 1522 still programmed from the LA regional so it just sat there with no connectivity for the first match. We reprogrammed it for the second practice session and the robot worked but was physically damage by a collision. I was not a witness to this match so I don't know the details. Our first match on Friday morning the robot started in teleop but lost communication almost intermediately for 42 seconds then lost again for another 40 seconds or so. End of match. Second time lost comm at the start of the match again the only movement I saw was our shooter motor trying to start then we lost comm. The FTA said it was for 40 seconds or so they suggested it was the Crio rebooting. I never saw the RSL light go off during any of the matches so I don't think the cRIO rebooted. I'm not sure what we replaced the next match and so on but we tried everything you did 3 radios , cRio twice, 12-5volt converter, PDB 3 times , basic source code, removed all other supposed power noise problems except the drive wheels. For our last 5 matches on Friday I watched from behind the FMS display. As soon as they reloaded the display with our match teams the display would show us as disconnected and dropping packets. They would wait till we intermittently showed green then start the match. Needless to say we lost comm every time at the start. Did I mention that Friday night when everybody was gone we ran OK? The order of replacing the different items are not clear to me as they weren't under my control. At this point I was just shaking my head and thinking how are we going to get this robot back together for St. Louis. On our 9th match we had replaced the digital side car?? and the cRio again along with the cards?? Now this is what I can say for certainty about this match. I'm sitting behind the FMS display again at the start of the match they bring our robot onto the field the display is updated to our current match. The display shows our robot dropped comm with packets dropping constantly. Two FTA's go to our robot do something (several minutes went by ) (queue dancing and loud music) They start to leave the arena and our robot shows connectivity and no dropped packets. Match starts our robot moves sort of Jags miss wired ball pick up set to the compressors pwm signal but we move the whole match with very few packets lost. I asked the FTA what he did and he said they connected the driver station to the robot when they saw no connectivity and it connected ok through a cable. But when he disconnected the Driver Station the radio would not connect to the field so he powered off the robot to rebooted the robot. When it came back up we have connectivity for the entire match. Next match we make sure we do not connect the robot to our driver station by cable or if we do we power the robot off then on and so on our last match we ran again not perfect because of the code we had running but we ran the whole match with few lost packets. The only thing I can say is that you shouldn't connect the DS to the robot prior to going onto the field with out turning the robot off afterward. We will check our radios and crio at the school when I get a chance. Trying to find the original culprit. BTW this is only our second year and are just starting to learn about the drivers log etc.