Team Fusion #364, Bayou Regional, FMS Woes

[Joe Ross]

Originally Posted by RyanN

We had trouble maintaining connection to FMS during the pre-match announcements. Why? No EMI, no vibrations, and we know our power is good from testing all the power and connections in the pit. At this point, all of our hardware had been replaced. Our power test consisted of probing voltages when the robot was still, everything came up clean. We then enabled teleoperated, shook, kicked, dropped, rattled the whole robot, and saw no loss of communication, no dropped packets, and no spikes in latency. I'm confident that the power on our robot is in good shape after when I did to the robot during testing. In no case would our robot be subject to the abuse I gave it.

Even more troubling is why it worked Friday night when no one else was in the venue, and why it didn't work Saturday morning when we were still the only one on the field powered up.

So yea, why would we connect and drop before the match even began? It seriously sounds like a crowded WiFi channel or a resource conflict on the FMS side of things.

Your symptoms sound exactly like ours, except that our problems did not stop when we stayed late on Friday night after everyone else left.

We did monitor the cRIO and radio power with a DMM when the dropouts occurred, and there was no change. However, replacing the power distribution board did fix our problem. I can't explain it, but it worked.

[RyanN]

Originally Posted by b-rant

Ryan, did you try switching the wireless channel or downloading a utility to see what wireless channels were being used around a field in a typical match? If switching the channel doesn't work then we just need to start building a Faraday cage around the field I hate that you guys had to go through that this year since the robot looked like a mean machine. Are you guys going to be able to attend any other regionals or was that it?

-Brantley
Team Fusion Captain 2007

We talked about it, but the FMS is so particular about everything. They have it all setup with a kiosk, and the teams aren't supposed to change anything on the router. We couldn't get FTA to budge on much of anything. The fact that it was connecting sometimes was good enough for them to say that the problem was ours and not theirs.

I wish I would have had software to check how busy the channels were. I feel pretty certain our issue was something to do with conflicting resources on the network. That's the only thing that made sense to me why it worked Friday night when no one was there, and didn't work the next morning.

The fact that it worked fine on Friday night, and that we're using such dumbed down code with no features, just basic drive code leads me to believe that the issue is not code related at all. Also, the fact that it runs everywhere but the playing field lets me know that the problem isn't code related.

We're talking with some people about trying to get pushed into the Lone Star Regional real fast.

[rbtying]

Despite our hopes and expectations following a full four and half minutes of connection in the morning, Team 846 was unable to maintain our connection to FMS throughout any of our nine matches.

In the morning, we had a few minutes on the field with FMS to test communications, along with another robot. Though things appeared to magically work following changes to task priority in the code, the same (deployed) code did not work in the actual match with six robots on the field.

It is probably useful to note that, according to FMS logs, less than 10% of matches at NYR ran without at least one robot dropping communications.

[wireties]

We had a few minor latency issues with the Bayou FMS on Friday morning but not after that. Team Fusion had a great robot. It is a travesty they did not get to play! They were definitely robbed.

I agree with the Fusion mentors on this thread, interference of some kind makes the most sense. But I have one (pretty weak) troubleshooting comment - was the radio always mounted in the same location? Did I see a statement about it being mounted "under a motor"? Did you try mounting it high and without any noisy active electronics in close proximity? Were the delays in communications several seconds or 1 minute plus?

Though you replaced the radio many times, did you replace the power and ethernet wiring?

[AllenGregoryIV]

Originally Posted by RyanN

We're talking with some people about trying to get pushed into the Lone Star Regional real fast.

I really hope this works out, we would love to have you come to Houston. Your robot looked fantastic and it's travesty we never got to see it at its full potential.

A program I have found that works well for viewing wireless channels and possible interference is called inSSIDer.

I really hope FIRST can find the cause of these communication issues. The power distribution board and the router configuration seem to be the two things that were never changed. I wonder if configuring your setup as a different team number would have resolved the problem.

[RyanN]

Originally Posted by AllenGregoryIV

I really hope this works out, we would love to have you come to Houston. Your robot looked fantastic and it's travesty we never got to see it at its full potential.

A program I have found that works well for viewing wireless channels and possible interference is called inSSIDer.

I really hope FIRST can find the cause of these communication issues. The power distribution board and the router configuration seem to be the two things that were never changed. I wonder if configuring your setup as a different team number would have resolved the problem.

We asked about another, but temporary number and they said they can't do it. The router was moved to the best location available on the robot. It was located under two 550 motors, but has now moved to an unpopulated area velcroed to some lexan above the crio, digital sidecar, and our solenoids.

[Jimmy the Kidd]

As inexperienced as I am, I still tried to the point of headaches to figure out what the problem could have been for us. I've never been so frustrated in my life, nor more disappointed. However, I felt like the decision we made regarding finals was the best thing we could have done. We'll see what the future holds, what with this being my last year as a student.

[Tom Line]

I feel for you guys. We've never missed as much of a regional as you did, but we missed the first 1/3 due to control issues in '09, and I know how horrible we felt. I can't imagine being in your situation.

I didn't see it listed anywhere, but I may have missed it: Have you guys gone out with a different driver station and no cypress board to see what happens?

[RyanN]

Originally Posted by Tom Line

I feel for you guys. We've never missed as much of a regional as you did, but we missed the first 1/3 due to control issues in '09, and I know how horrible we felt. I can't imagine being in your situation.

I didn't see it listed anywhere, but I may have missed it: Have you guys gone out with a different driver station and no cypress board to see what happens?

We went out with 3 different laptops as our driver station.

1. The old style classmate that we received in 2010.
2. A Dell M5040 (we personally don't like this one, and it restarted during our first match on Friday due to pending Windows Updates... go figure)
3. Late 2010 MacBook Pro (running Windows 7 natively)

All three laptops are fully capable of running the Driver Station software along with the Dashboard.

We have not tried running without the cypress board. It's an integral part of our robot design. Running without it would hinder our driver from running. The Cypress board shouldn't be the problem.

On Saturday, when we were desperate, we used the Classmate, with the default dashboard, and the dumbed down code that I've mentioned before. It was a stock FRC system.

Joe, we'll try replacing the PD board when we get a chance. As of right now, we're not registered for another regional. We're going to try to register, and as such, we don't want the students messing with the robot until a final decision has been made.

[Elizabeth Waters]

Yep, we had a very similar issue several times on the field at Bayou.

We never had any issues connecting to the field before the match began. BUT, after running autonomous, we'd still be "connected" but the driver station would display "No Robot Code." When this happened the first time during a practice match on Thursday, we waited to see if it would find the code; never did. The FTA suggested that next time we try to reboot the cRIO in the middle of the match and see what happened. So we listened: every time we "lost robot code" on the field (which happened at least 4 times over the course of Friday and Saturday), we would immediately reboot the cRIO. What would happen:

1) Auto works fine
2) Teleop begins: "No Robot Code"
3) Driver immediately reboots cRIO = lose connection with FMS temporarily
4) Regain connection within ~5/10 secs.
5) "No Robot Code" displays again
6) Regain comm ("find" code) with about 20 sec. left in match

So ultimately, reboting the cRIO worked for us (if I can really say that... We were still out most of the match, obviously). Did you guys ever try this?

But you're right, this still doesn't answer the question of why this occurred in the first place. Our losses were more sporradic than yours, so not sure why ours worked most of the time. When we talked to the FTA about this issue during quals, he blamed it on our robot, too (and you know, who knows, it very well may be the robot). When it happened again during elims, FTA said it still looked like FMS was fine; they said there were no issues on their end, and when the code was "lost" after auto, everything was still reading fine in relation to their connection with our bot. Who knows.

Hope someone can figure this out soon.... We're also attending Lone Star in a few weeks.

[RufflesRidge]

Originally Posted by blondie624

Yep, we had a very similar issue several times on the field at Bayou.

Hope someone can figure this out soon.... We're also attending Lone Star in a few weeks.

I don't think this sounds like the same issue based on Ryan's description of 364's problems. It sounds like your code is crashing or sucking up all the CPU.

I would use the Practice match feature on the Driver Station and watch your CPU and memory on the Charts tab of the Driver Station.

I would also look very carefully at all image processing code and make sure you are limiting the number of particles you are processing and catching any exceptions or handling any errors that may be generated including return values of Null from the various vision pieces.

[otherguy]

2168 seemed to have this problem at NYC.

We never had an issue, period. Not for the entire competition until finals. We went 8-0-0, ranked #1, and had the highest teleop points. Our robot works, and we showed that. That's what makes this so unbelievably frustrating.

During our first semi-finals match... all three robots lost communication on the field. 2168 and 329 came back up but 1676 was disabled all match. The blue team did not see such an event. One of our mentors spent 10 minutes with the FMS manager diagnosing what was causing the problem. He said: "There was nothing they would do but it was very evident from the data the FMS collects that all robots were having 200 - 700ms delays."

1676 changed their bridge and Ethernet cable, and 329 deployed new code.
2168 verified wiring and swapped out one of our Jags which seemed to be failed on the drive train*** and the battery that we used during that match was getting way too low ~7V during the match. It was marked as good. (It is now marked as failed). We checked frantically for shorts but our chassis (as it had been all competition) showed infinite resistance to ground and pwr. The battery cable to the andersen connector was noticeably warm to the touch, andersen to the PD board was not however...

***Later we found, during pit testing, that the snap action breaker at that position seemed to be failing open. We have not yet tested the Jag we thought to be failed, it may have been fine, just not receiving power.

ultimately during our second semi-final match we had intermittent comms again. The lift motors were non-operational and since there was a loss of comm issues the drivetrain was not responsive.

Something to note, our team uses victors on the lift (pwm to digital sidecar on slot 2 of 4 slot crio). All other motors are CAN Jaguars connected to 4slot crio via DB9 connection. All 6 Jaguars remained operation during the matches. The victors were apparently not receving commands, however I was on the field right next to our robot and could see the light on the victors were solid, so PWM signals were apparently being sent?

All testing between matches on the side of the field showed all systems being fully functional. No faults with CAN traffic.

After jerking around in semi-2 we brought the robot back to the pit to try to figure out what could have been going wrong. Our operator noted that our button box (psoc3 FIRSTouch IO Module) was acting "weird". Typically in the IO tab on the driver station we should see all 16 Digital IOs pulled high. All indicators were extinguished, and not changing state with button presses, every now and then random ones would flicker on and off. Analog channels appeared to be responding properly to potentiometer changes.
We're using the classmate for the driverstation. It had not been restarted since the semi match ended.

Wiring for the board is all potted in glue, so its unlikely that it would have failed randomly. We checked wiring however and wiggling the usb connection at the psoc seemed to cause a fluttering of indications on the driver station. So we thought we had a wiring problem. I didn't believe we had a random failure of that board (The usb cable is knotted before leaving the box to prevent any damage inside to cabling), so I plugged it into my personal laptop and opened the driver station. Everything worked flawlessly. I plugged it back into the classmate and the same issue presented itself as before.
When the power cable was plugged into the classmate the indicators fluttering seemed to be more erradic which led me to think there was a noise issue on the classmate usb port or the hub we are using.
Restarting classmate (including battery removal) resolved the issue however, and it couldn't be replicated again.

We were able to replicate the field issue (the victors not responding - no ball lift) once in our pit, but over an hour of testing after that revealed everything to be operational.

We've come up with a list of items to check/replace at our next regional. PD board is at the top of my list, but that doesn't explain the weirdness we saw with the PSOC.

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

Hopefully we can shed some light on what's going on here and come to a resolution. Good luck to everyone in fixing the issues on your bots. Keep us updated.

[Greg McKaskle]

Team 624, I may know what your issue is. I'm still looking into a detailed fix, but I'd like someone from the team, ideally a mentor, to PM me. Sorry if this sounds secretive, but I saw something at WPI, and I just tracked it down Saturday morning. I want to get a few more data points at this point. And no, this isn't causing all field comms issues.

Greg McKaskle

[MikeZ]

Greg

I will PM you later this morning. Thanks

[thefro526]

Originally Posted by otherguy

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

In 2010, 816 had numerous issues with the PSOC board - we'd randomly die out mid match, lose one side of our drive the next, and then lose the entire drive but retain all of our aux functions the next. Our 'easy solution' was to unplug the PSOC and replace the custom part of the DS with a game pad. After the season we replaced the PSOC with a CCI from eStop and never had any issues with that robot or our 2011 machine: http://estoprobotics.com/estore/inde...d&productId=33