Team Fusion #364, Bayou Regional, FMS Woes

Technical Programming
1

I’m not sure how many people have had issues at the Bayou Regional, but we haven’t been able to have a single good match at the Bayou where we haven’t disconnected from FMS. They verified that our router remains connected, but they are saying our cRIO keeps dropping off.

We have replaced everything, and I mean everything regarding the control system. Electrically we’re good. Our best guess is that the cRIO is crashing mid match.

Here’s the thing we cannot get past. Our code works perfectly tethered and on the practice field using wireless. We can beat the heck out of our robot, and it remains connected. CPU usage hovers around 40%, memory around 40%, we have tons of flash space free. When we put our robot on the field, we’re always the last to connect to the FMS, we’ve had it drop and reconnect multiple times while the field is being setup. The field management bypassed us one match and told us we couldn’t play because they couldn’t keep a connection.

My question is why? I know we weren’t the only ones with the issue today. Kryptonite had the same problem during the same match we were in one time. Another team came to talk to FTA after the award ceremony about the same issue. FTA still claims that they don’t see any issues with their system. Maybe they don’t see any, but if our code runs perfectly everywhere but the field, that makes me believe there is a problem on the side of the code we cannot touch (cRIO Image, and FMS).

We can sometimes make it through autonomous, and during the match, our robot will come in and out of communication. We finally remained connected for at least half match in our last match of the day, and we did great.

I hate to see our kids’ work that they spent 6 hard weeks working on go to waste because of a dumb bug or glitch with the cRIO image.

I know there was a team update on ‘lightening’ code, but we’re not running anything more complicated that what we ran in 2008 on a PIC18 microcontroller, which is way less powerful than the cRIO we’re given today. We tried lightening the code during the day between matches, and we finally finished right at 7PM today. We have one match left, and it’s the only match we can try to redeem ourselves. We have no closed loop control, and I’m calling it our dumb code, because all it does is take joystick output and apply it to motors. Nothing neat, no safety interlocks, no special features. All of the hard work we worked on is useless now because of a problem that we cannot diagnose outside of the FMS system.

Honestly, I’m exhausted. We’ve spent 2 days now working on this problem. Our CTA, MJ, looked through all of our code, witnessed it working in the pit, and agreed with us that our code works. We have sat in every match so far. We have a very capable robot, and we know it. It’s so hard to accept that we basically missed every match this year for a 12 year old team.

Thanks,
Ryan N

I would also like to add that I have over a year of professional LabVIEW experience. Our code can be found here: http://www.chiefdelphi.com/forums/showthread.php?t=103902

2

I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?

Is your radio mounted near motors, and anything else that could cause interference?

Is it buried deep within the robot, possibly shielded from the wireless signal?

Was the router used a lot last year? A bit beaten up? Could any foreign material have fallen into it? Could it just be dirty inside?

Is the power connector to the radio nice and tight? Hot glue, rubber bands or velcro straps could help here.

These are all a bit of a reach, but at this point, I’m sure you’re willing to try anything!

3

Dear Ryan

First off, I’m really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.

I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.

We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven’t tried it, although I’m sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year! :wink:

Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I’ll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn’t something to do with the crio rebooting. We did notice that we were dropping packets.

The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we’ll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.

Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.

Once again, I’m really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!

4

Is the router clear of motors and things that might interfere with it? We had to move ours because FMS didn’t like the fact we put it low, near the drive motors.
Try a new cRIO? Image it?
Not really sure what it could be just throwing out some ideas. Issues like this always suck, I remember FMS costing us a match or two last year.
Good luck to your team, and I hope you guys get it sorted.

5

Our radio is connected below our shooter motors, but we’re losing connection while disabled as well, so this is not caused by EMF.

I will update tomorrow after our match if we run or not. I have a bad feeling that it has been the code the entire time. No reason the code is failing, just that it’s too complicated for the FMS to handle for some dumb reason. I know it doesn’t make sense, but there’s nothing else I can think off as to why it works great everywhere except the field.

6

I’m also at a loss for what could be the cause of your communications issue. I’d be curoious though if you’re able to replicate the same issue in the pits while tethered and running FMS light?

you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx

direct link to the download: http://208.82.132.46/download/fmslight.setup.msi

i’ve had good luck with this version on windows 7 but not some of the others… i don’t know if it will exhibit the same issues when tethered and with this version of FMS, but it might be a worthwhile test… (quick setup = install, set your ether ip to 10.0.0.5, open FMS, click the big reset button(seems to need it every time you open it fresh), then put your team number on one spot and bypass the others, DS should show FMS control and you should be able to start the match from the FMS panel)

if by chance you ARE able to replicate with FMS tethered in your pit, then immediately try a simple example code if that has issues then it should be something electrical/physical still… if that code is fine but your own code is not - you know where the problem must lie…

hope that helps - good luck and keep us posted.

  • Jon
7

Make sure your not getting something called a watchdog error. There is a way to disable it in the code. I am not by any means a programmer but this has happened to my team before. The symptoms you describe are pretty accurate to what was happening with us. Good luck.

8

I’ve been seeing these symptoms alot, each comes with a smorgasbord of possible causes, and any number of eventual outcomes. There were many when the system was new to us in '09, but there seems to be an increase this year.

One thing I’ve seen in common is that the electronics were mounted to polycarb sheet. Both '09 and this year the robots are like vandergraph generators (fast moving belts within an isolated shell). Is it possible for a polycarb surface to build up static until the conditions are ripe for it to discharge?

9

It sounds like we might be having a similar issue. We’ve extensively replaced components and code, without a solution (yet). http://www.chiefdelphi.com/forums/showthread.php?threadid=104713

10

Can I get DS log files from each of you emailed to me. Also can you include a description of the cRIO, the DLink radio particulars including the firmware version. I have your code Ryan. I’m not sure if that is having any impact.

I was at a week one and two regional and haven’t seen this. Comms issues have typically been firewall, IP, or electrical issues. I’ve also seen the issues go away when the radio was reprogrammed or swapped out. I wish I knew what was causing those.

PM me for my gmail account, though you can probably guess what it is.

Greg McKaskle

11

We’ve tried 3 radios, all rev A. Two with firmware version 1.21 (one was ours and one from spare parts) and one with firmware version 1.4 (purchased last month). We’ve tried 2 cRios, both have been 4 slot cRios (no room for 8 slot).

12

At Rutgers our team, along with 75 and 41 all had a similar problem to that described above, except that none of us could connect to the field at all.

Long story short, the automatic radio configuration system was not configuring the radio properly. When we configured it manually (with the explicit approval of the FTA) we were good.

Perhaps checking the radio configuration for oddnesses might help?

13

Do you know what setting was not good?

Greg McKaskle

14

And also, can we get a breakdown of what the exact WORKING settings were when you manually re-configured?

These settings will be a good reference for us in the future.

15

We had issues like these last year and replacing the driver station laptop fixed it.

We, to this day, are still trying to figure out why.

16

We ran our robot for 20 minutes in the practice field wirelessly. Made 30/30 shots, and it ran flawlessly. Right in front of one of the FTA. I asked him what he thought and he blamed the robot still.

I’m sorry, we worked for 3 days, completely replaced all the control system, replaced our code with basic drive code. It always worked out of the pit, but never on the field.

I call bullcrap on it being us. I’m an experienced LabVIEW programmer, 4th year computer engineering student, and very knowledgeable in electrical. Our robot is wired and configured properly.

FMS cost us every single match. We never ran a single match without disconnection. We would disconnect at any time. Before the match, during autonomous, or during teleop.

I hope FIRST figures out what’s going on, because there is nothing more depressing in FIRST to see your robot go on the field knowing it’s not going to work.

17

Maybe totally unrelated, but 176 was immobile for long stretches in finals 2 and 3 at Montreal. They worked in autonomous, then stopped dead for the rest of finals 2; then they worked for autonomous in finals 3 and stopped again for about 20 s.

One of their alliance partners was also having some trouble moving around.

(I’m not in Montreal, so I don’t know if there were other underlying reasons.)

18

When we had the symptom, we could connect to the radio using a browser but there were no settings. Noting, like a broken web page. Weird. We hard-reset the radio and could see settings (but they were default, of course). We compared to a good radio and got the WEP code, entered it all manually, and it worked.

We took our second radio, hard reset it, and got it programmed, and it was good, we could see the settings and all. Tried it in our next match, and both radios were OK after that. Go figure.

19

The links to FMSlight don’t seem to be working does anyone have new ones? I would love to test our control system before our event in two weeks.

20

The difference between the practice field and the competition field is more than just the presence of FMS. The quality of the wireless signal is going to be very different. Minor RFI issues on your robot, such as noisy power or speed controllers mounted too near the D-Link, have the potential to disrupt match communication without being bad enough to show up during practice.

If you have the D-Link mounted adjacent to anything electical, try separating them. I’ve seen that fix something that sounds like your symptoms on one team. If you get a chance, try looking at the 5v power going to the D-Link using an oscilloscope. Replacing the Power Distribution Board fixed the same symptoms on another team.