FMS/cRIO comm issue

[sdaustin]

We have an issue with our cRIO – FMS is unable to establish communications with it under any circumstances. The cRIO works fine in all other configurations. With the help of FIRST and National Instruments people (details below), we believe that there is something about the way that the FMS communicates with the cRIO that is exposing a hardware defect that is not otherwise exposed.

Therefore, I need to get in touch with someone that understands the intricate details of the low-level implementation of FMS.

The Details
We had to forfeit our first five qualification matches (and put our alliance partners at a significant disadvantage) at the Kettering District competition because our cRIO would not work under the FMS - it could not connect to the FMS & vice versa. It worked fine hard-wired or via wireless using the onboard DLink AP. We checked everything that the Lead Robot Inspector (Mike Martus), the National Instruments rep onsite (Andrew Watchorn), and the FIRST Technical Advisor (I don’t have his name), could think of, but no luck. Their consensus opinion was that FMS was exposing a cRIO defect. We exchanged ours for a loaner cRIO and everything worked fine.

After the event, I opened a support case with National Instruments, and it got assigned to a support rep (Joe) who was himself a FIRST mentor for the past eight years. After several phone calls and running through every possible thing he could think of to reproduce the problem in our shop, we went to the Waterford district competition the next weekend, hoping that the problem was resolved. Luckily, we were able to get our robot on the field before the opening ceremony to test it. The exact same problem still occurred. We replace the cRIO with a loaner and the problem went away, again.

But we still didn’t have a reproducible failure scenario. In spite of this, NI authorized an RMA for us to send the unit in for repair. And now, of course, they are unable to reproduce the problem. Their diagnostics can’t find anything wrong with our cRIO and so they want to send it back unless I can give them some situation under which they can reproduce the problem.

Unfortunately, the only way to do so is on an actual field at a competition, which is not a viable option because we’ve already gone to both of our district competitions and are done for the season.

So I either need a way to run the FMS in our shop or to get in touch with someone that can give me more details of how it is implemented in the hopes that we can create a reproducible scenario.

Any suggestions?


Thanks,
Scott Austin
Coach, Team 3667, The PoHo RoBos

[plnyyanks]

What image is your cRIO running? Currently, only versions 28 and 29 are guaranteed to work with the FMS.

[jcbc]

Did you try the procedure given in Team Update #17 for simulating the FMS? We haven't tried it yet, but we plan on trying it with our practice bot before going to our next regional.

[Radical Pi]

It is possible to get a semi-FMS running in your shop, although it is extremely different from what is used on the field this year. It's a stripped-down version of the 2009 FMS which as far as I can tell from FIRST's site is still compatible with the classmates. It can be found at the bottom of the page here

I'm obviously not an expert or anything on how the FMS works internally, but here's my impression of how the system works. When you connect your DS to the field, the FMS detects it and starts broadcasting commands, and the DS returns status data. This is the "FMS Connected" state shown on the main tab. Once the robot establishes a connection, it talks with the DS. As far as I know, there is no direct FMS-cRIO communication, other than the DS packets passing through the main FMS box on their way to the field router. No idea what happens internally there.

If you're able to reproduce the problem on FMS Light, I'd suggest getting in contact with FIRST HQ and talk with the writer of the FMS software. They'd have much better debugging experience.

[Matt Krass]

I would assume with that many bright minds in one spot the obvious stuff (cRio Image version and such) is probably not the cause. I'd be curious about putting the malfunctioning cRio next to a working one, each on it's own wifi bridge, etc, with an FMS system and running a packet sniffer on each bridge. Run the same code, same everything, and see if any particular difference jumps out. Unfortunately this is probably isn't very feasible, but I'm at a loss as to what else could be done to clearly identify the difference.

Of course, my assumption could be wrong, and it could be something remarkably simple

Matt

[Kevin Sevcik]

I'm kinda voting the packet sniffer method as well. As Radical Pi stated, the FMS never talks directly to the cRIO, only the DS. There's firewalling and etc. happening through all the field networking equipment, however. That seems like the only place likely to cause this problem. If Kettering and Waterford used the same field and networking hardware, that'd certainly be suspicious. There might be something on the network with a stray setting that targeted your particular cRIO. I know you can't test things anymore, but I'm curious of you did any/all of the following in all the testing:
Pinged the cRIO from the DS
Pinged the Bridge from the DS
Port scanned the cRIO from the DS
Monitored packets from the DS to the cRIO
Tried your cRIO with a different robot or team number
Tried your cRIO with a different Bridge

Basically, it seems likely to be a networking hardware issue somewhere, as opposed to something physically wrong with the cRIO. What would be extra interesting is if you could get your cRIO to the Michigan Championship, along with someone(s) clever enough to check the network traffic etc. Then there might be a chance of pinning the problem down for good and all.

[Radical Pi]

Originally Posted by Kevin Sevcik

If Kettering and Waterford used the same field and networking hardware, that'd certainly be suspicious.

That's a good point. The listing of what fields are going where (sorry I don't have a link anymore) shows Kettering and Waterford both using the same field hardware. If you do make it to MSC, that would be a good chance to check if it's just one field as it's using the other Michigan field.

[sdaustin]

Thanks for everyone's very quick and pertinent responses. To answer as many questions as possible:

- We were running cRIO image 28. The cRIO was imaged twice at Kettering; once by the Lead Robot Inspector and once by the NI rep. A match was attempted between each imaging.
- We tried different network cables from the DLink to the cRIO onboard. One was placed in different location from the original - it's unlikely that something else onboard was injecting noise into the cable.
- At Kettering, the FTA loaded the settings onto our DLink twice. We also tried two of their loaner DLinks which they had configured.
- We were running DS 2.27.11.00 on our Classmate. We also tried a loaner Classmate for one of our matches. It didn't help.
- We formatted the onboard cRIO flash "hard drive" and reinstalled image 28 between competitions. To my knowledge, image 29 was not yet available at that point.
- We had run the Team Update #17 procedure between competitions.

I too am thinking that a packet sniffer and side-by-side comparison under FMS is the next logical step. Unfortunately, the cRIO is in NI's hands for now, so that will have to wait.

We're done with competitions for this year. Now we're focused on things like giving demos of the robot to the Board of Education and middle schools, participating in the ACS Relay for Life, etc.

[Matt Krass]

Originally Posted by sdaustin

I too am thinking that a packet sniffer and side-by-side comparison under FMS is the next logical step. Unfortunately, the cRIO is in NI's hands for now, so that will have to wait.

NI will do this test, or a better one they most likely have available, this is their baby and they're going to be mighty unhappy it's throwing a temper tantrum.

Hopefully this was an isolated incident, though I'd be curious to know what the cause was for future reference.

Matt

[anyheck]

MAC address filtering in the firewall on that field for some reason?

No idea why they would have that.

Just throwing it out, as it's directly tied to the hardware.

[Al Skierkiewicz]

sd,
I have to ask, was there a stable voltage display on the Classmate dashboard? Although NI should have seen something by now, I have two teams this year reports similar problems. One had a bad Crio pin in slot 1 and one had a bad solder job on the analog bumper on module 1. Do you remember the action of the RSL when on the field? Also, do you run an autonomous program and did it run that OK?

[sdaustin]

MAC filtering probably isn't the issue. I'm told the FMS does not send any traffic directly to the cRIO. It's all commands to the DS, which then in turn sends commands to the cRIO. We tried two different DLinks and a different DS.

I can't recall exactly what was displayed on the DS, other than that the uppermost indicator in the lower-left pane ("Comm", I believe) simply never turned green running under FMS. Without Comm, the robot battery voltage was probably not being displayed. The DS was usually on AC. When we swapped out the cRIO, we continued to use our existing modules and didn't change any of the wiring to/from them. Given that everything worked fine if the DS was hardwired directly to the cRIO or thru the DLink, or if we used a different cRIO, I think that rules out the battery and modules.

What's the RSL?

Autonomous wasn't an issue for the first five matches -we carried the robot off the field before they started. My recollection is that it was disabled for the first few matches w/ the new cRIO, then we enabled it. It was enabled for all of the Waterford competition and by about the 6th match our kids had it working very well, following the lines & hanging an ubertube. Woo Hoo!

This smells like an OSI Layer 1 or 2 network issue in the cRIO ethernet port to me.

Anyway, I have a high level of confidence that the NI application engineer working on this will get it resolved to our satisfaction soon. He has been truly awesome.

[jtdowney]

Originally Posted by sdaustin

What's the RSL?

Robot Signal Light, it is the large orange Allen-Bradley light that is wired into the Digital Side Car that connects through slot 4.

[anyheck]

Originally Posted by sdaustin

MAC filtering probably isn't the issue. I'm told the FMS does not send any traffic directly to the cRIO. It's all commands to the DS, which then in turn sends commands to the cRIO. We tried two different DLinks and a different DS.

I was suggesting that because no traffic would pass to the cRIO regardless of the source of the traffic, if this was set up in the router or firewall somewhere.

[Mark McLeod]

If you think it was an issue with the cRIO Ethernet port, then what were the port lights displaying while the error occurred?

There is a fault that exhibits itself as solid red & amber lights on the cRIO port where it fails to communicate with anyone. It's apparently a lockout condition between the cRIO and the DLink, possibly related to the ARP. Everyone's on but nobody is talking.

Power cycling does nothing to help (it's actually counter-productive in this case), because the timing is such that the order the devices come up seems to cause the lockout.
I've seen it fixed by a simple disconnect/reconnect of the cRIO Ethernet cable followed by a minute of patience. Sometimes a reset of the cRIO alone will clear the condition, but only sometimes.

A sniffer would certainly help diagnose the issue.