Weird Socket Problem with Arduino & cRio

[codes02]

What symptoms are you observing from the cRIO that causes the appearance that it has "locked up"?

Are you certain that the Arduino is still outputing data over ethernet? (relevance of this depends on the above q. From your description about "not actively reading" on the cRIO, this doesn't seem a likely issue, but do check).
On checking this: connect a ethernet hub between the cRIO and the Arduino. Plug in a computer to this hub as well. Use wireshark to watch packets.

Also: what is the network setup like? For example: are you running the Arduino's ethernet through the Router, or directly?

[taichichuan]

Originally Posted by codes02

What symptoms are you observing from the cRIO that causes the appearance that it has "locked up"?

Are you certain that the Arduino is still outputing data over ethernet? (relevance of this depends on the above q. From your description about "not actively reading" on the cRIO, this doesn't seem a likely issue, but do check).
On checking this: connect a ethernet hub between the cRIO and the Arduino. Plug in a computer to this hub as well. Use wireshark to watch packets.

Also: what is the network setup like? For example: are you running the Arduino's ethernet through the Router, or directly?

In the failure mode, the cRio completely drops off of the network. The communications "led" on the driver station goes red, the net console stops responding and the only way out is to reboot the cRio. I can monitor the link with Wireshark and see that the packets are still going out over the wire. The Arduino is hooked up to the Dlink router as is the cRio.

The Arduino is outputting a packet once a second that contains a message ID and the RPM count in a null-terminated packet of about 8 characters in length. This is a directed unicast specifically to the cRio's IP address. I've tried both the UDP port for DS to cRio and a generic port (8888) with similar results. Of course, the Arduino starts outputting packets as soon as it's powered up.

We find that if we create the socket in the robot constructor and then just let it sit (don't enable anything like it would be on the field waiting for the FMS to get going), that the system will fail in typically 2-3 minutes. However, if we have a piece of code that consumes the messages, everything works as it should.

I've been a VxWorks user since 1987 and even helped write sections of the O/S back in the day. I've never seen this sort of behavior in the base O/S. So, It must be something FIRST/wpilib related. If anyone can shed some light on this, I'd be really interested try an understand what's happening. I'm hoping that I can create a similar piece of code on my Linux box to generate packets and send them to the cRio to test to see if it has the same failure mode.

TIA,

Mike

[jhersh]

Originally Posted by taichichuan

I've been a VxWorks user since 1987 and even helped write sections of the O/S back in the day. I've never seen this sort of behavior in the base O/S. So, It must be something FIRST/wpilib related. If anyone can shed some light on this, I'd be really interested try an understand what's happening. I'm hoping that I can create a similar piece of code on my Linux box to generate packets and send them to the cRio to test to see if it has the same failure mode.

Does this occur only if you open a socket on that port and never read, or also if you never open the socket at all?

BTW, the DS to cRIO port is only for crossing the field network. You should use ports outside the FRC range (as defined in the rules) for on-robot traffic.

-Joe

[taichichuan]

Originally Posted by jhersh

Does this occur only if you open a socket on that port and never read, or also if you never open the socket at all?

BTW, the DS to cRIO port is only for crossing the field network. You should use ports outside the FRC range (as defined in the rules) for on-robot traffic.

-Joe

Hi Joe,

This is Mike Anderson. We've met several times at ESC. I'm the software mentor for Team 116. In any case, I've been able to duplicate the failure from Linux as well. I'm attaching the wireshark output, the code for the cRio that fails and the code under Linux that causes the failure. It generally fails within 70 packets or so causing the cRio to die.

So, compile the UDPtest and load it to the cRio.
Start the client on linux (set to talk to 10.1.16.2 at port 1929).
Within 70 or so packets, the cRio will die a horrible death. No diagnostic output on the net console. It just dies.

To compile the client code under Linux, use "gcc -o client client.c"
The client.bin.txt is the compiled Linux IA32 binary.

TIA,

Mike

[Radical Pi]

Hi,

While we haven't seen the same problem as you, we do have something similar to your setup running on our robot. We do our vision processing on an external board, which sends the results via UDP to the cRIO. We have our receiver running in a separate thread, and have had no issues with it. You can see our implementation here.

Good Luck

[taichichuan]

Originally Posted by Radical Pi

Hi,

While we haven't seen the same problem as you, we do have something similar to your setup running on our robot. We do our vision processing on an external board, which sends the results via UDP to the cRIO. We have our receiver running in a separate thread, and have had no issues with it. You can see our implementation here.

Good Luck

Hi Radical,

Yeah, I think we've stumbled across the same issue here. Thanks for the code reference. It looks like you're using a Beagleboard with USB camera and OpenCV? How's that working out for you? We had considered that approach as well, but ran out of time to make it happen. With St Louis next week, we're just working to make sure that we're as stable as possible. This UDP problem is yet another little surprise that we didn't anticipate.

Thanks for your help,

Mike

[Radical Pi]

Originally Posted by taichichuan

Hi Radical,

Yeah, I think we've stumbled across the same issue here. Thanks for the code reference. It looks like you're using a Beagleboard with USB camera and OpenCV? How's that working out for you? We had considered that approach as well, but ran out of time to make it happen. With St Louis next week, we're just working to make sure that we're as stable as possible. This UDP problem is yet another little surprise that we didn't anticipate.

Thanks for your help,

Mike

Correct on all counts. The USB camera is actually a kinect. It's worked fairly well overall (we can pinpoint our field position down to a 1/3 foot with it), though the kinect itself has provided our own series of surprises.

We'll be at St Louis as well, so if you need any help with this, feel free to drop by 639's pit