Weird Socket Problem with Arduino & cRio

[taichichuan]

Hi Fellow FIRSTers,

OK, we're trying to use an Arduino to sample the speed of our shooter and the angle of the ball guide. We're using Ethernet on the Arduino because the cRio II doesn't lend itself to much of anything else. FWIW, we're using C++ for our code.

The Arduino is working fine and outputting packets on the local network segment on the 'bot. We're using UDP (port 1130) because there's no need to have a connection given how often the updates come out of the Arduino. If we miss one, there'll be another coming along soon.

Our problem is that if the Arduino is outputting packets and the cRio isn't actively reading them, then the cRio will eventually lock up. It's acting like we're filling up some sort of buffer that's causing the robot code to hang because it's not being read. But, I've never seen this behavior with VxWorks before. Normally, if there's nothing reading a UDP port, the packet buffer fills up and then the packets just get dropped.

Can anyone shed some light on the issue? Is this something that was changed in VxWorks to accommodate FIRST? Is there anyone else who is using Ethernet from an Arduino to talk to the cRio? Any enlightenment would be greatly appreciated.

TIA,

Mike

[virtuald]

Not sure how portable your code is, but one idea might be to run it on a PC and see if it exhibits the same problems there.

[jhersh]

I'm not aware of anything that was changed for FRC that would have anything to do with this issue. Is it possible that something else is using that port? Have you tried using a different port?

[codes02]

What symptoms are you observing from the cRIO that causes the appearance that it has "locked up"?

Are you certain that the Arduino is still outputing data over ethernet? (relevance of this depends on the above q. From your description about "not actively reading" on the cRIO, this doesn't seem a likely issue, but do check).
On checking this: connect a ethernet hub between the cRIO and the Arduino. Plug in a computer to this hub as well. Use wireshark to watch packets.

Also: what is the network setup like? For example: are you running the Arduino's ethernet through the Router, or directly?

[taichichuan]

Originally Posted by codes02

What symptoms are you observing from the cRIO that causes the appearance that it has "locked up"?

Are you certain that the Arduino is still outputing data over ethernet? (relevance of this depends on the above q. From your description about "not actively reading" on the cRIO, this doesn't seem a likely issue, but do check).
On checking this: connect a ethernet hub between the cRIO and the Arduino. Plug in a computer to this hub as well. Use wireshark to watch packets.

Also: what is the network setup like? For example: are you running the Arduino's ethernet through the Router, or directly?

In the failure mode, the cRio completely drops off of the network. The communications "led" on the driver station goes red, the net console stops responding and the only way out is to reboot the cRio. I can monitor the link with Wireshark and see that the packets are still going out over the wire. The Arduino is hooked up to the Dlink router as is the cRio.

The Arduino is outputting a packet once a second that contains a message ID and the RPM count in a null-terminated packet of about 8 characters in length. This is a directed unicast specifically to the cRio's IP address. I've tried both the UDP port for DS to cRio and a generic port (8888) with similar results. Of course, the Arduino starts outputting packets as soon as it's powered up.

We find that if we create the socket in the robot constructor and then just let it sit (don't enable anything like it would be on the field waiting for the FMS to get going), that the system will fail in typically 2-3 minutes. However, if we have a piece of code that consumes the messages, everything works as it should.

I've been a VxWorks user since 1987 and even helped write sections of the O/S back in the day. I've never seen this sort of behavior in the base O/S. So, It must be something FIRST/wpilib related. If anyone can shed some light on this, I'd be really interested try an understand what's happening. I'm hoping that I can create a similar piece of code on my Linux box to generate packets and send them to the cRio to test to see if it has the same failure mode.

TIA,

Mike

[jhersh]

Originally Posted by taichichuan

I've been a VxWorks user since 1987 and even helped write sections of the O/S back in the day. I've never seen this sort of behavior in the base O/S. So, It must be something FIRST/wpilib related. If anyone can shed some light on this, I'd be really interested try an understand what's happening. I'm hoping that I can create a similar piece of code on my Linux box to generate packets and send them to the cRio to test to see if it has the same failure mode.

Does this occur only if you open a socket on that port and never read, or also if you never open the socket at all?

BTW, the DS to cRIO port is only for crossing the field network. You should use ports outside the FRC range (as defined in the rules) for on-robot traffic.

-Joe

[taichichuan]

Originally Posted by jhersh

Does this occur only if you open a socket on that port and never read, or also if you never open the socket at all?

BTW, the DS to cRIO port is only for crossing the field network. You should use ports outside the FRC range (as defined in the rules) for on-robot traffic.

-Joe

Hi Joe,

This is Mike Anderson. We've met several times at ESC. I'm the software mentor for Team 116. In any case, I've been able to duplicate the failure from Linux as well. I'm attaching the wireshark output, the code for the cRio that fails and the code under Linux that causes the failure. It generally fails within 70 packets or so causing the cRio to die.

So, compile the UDPtest and load it to the cRio.
Start the client on linux (set to talk to 10.1.16.2 at port 1929).
Within 70 or so packets, the cRio will die a horrible death. No diagnostic output on the net console. It just dies.

To compile the client code under Linux, use "gcc -o client client.c"
The client.bin.txt is the compiled Linux IA32 binary.

TIA,

Mike

[Radical Pi]

Hi,

While we haven't seen the same problem as you, we do have something similar to your setup running on our robot. We do our vision processing on an external board, which sends the results via UDP to the cRIO. We have our receiver running in a separate thread, and have had no issues with it. You can see our implementation here.

Good Luck

[taichichuan]

Originally Posted by Radical Pi

Hi,

While we haven't seen the same problem as you, we do have something similar to your setup running on our robot. We do our vision processing on an external board, which sends the results via UDP to the cRIO. We have our receiver running in a separate thread, and have had no issues with it. You can see our implementation here.

Good Luck

Hi Radical,

Yeah, I think we've stumbled across the same issue here. Thanks for the code reference. It looks like you're using a Beagleboard with USB camera and OpenCV? How's that working out for you? We had considered that approach as well, but ran out of time to make it happen. With St Louis next week, we're just working to make sure that we're as stable as possible. This UDP problem is yet another little surprise that we didn't anticipate.

Thanks for your help,

Mike

[Radical Pi]

Originally Posted by taichichuan

Hi Radical,

Yeah, I think we've stumbled across the same issue here. Thanks for the code reference. It looks like you're using a Beagleboard with USB camera and OpenCV? How's that working out for you? We had considered that approach as well, but ran out of time to make it happen. With St Louis next week, we're just working to make sure that we're as stable as possible. This UDP problem is yet another little surprise that we didn't anticipate.

Thanks for your help,

Mike

Correct on all counts. The USB camera is actually a kinect. It's worked fairly well overall (we can pinpoint our field position down to a 1/3 foot with it), though the kinect itself has provided our own series of surprises.

We'll be at St Louis as well, so if you need any help with this, feel free to drop by 639's pit