Image processing on the driver station laptop

[Chris Hibner]

I ran a test using UDP two days ago and I wasn't very happy with the results - perhaps one of the NI guys could say if what I saw is typical or not.

I set up a test program to communicate via UDP from a laptop to the cRIO. The laptop was set up to countinuously count up and the value of the counter was passed to the cRIO via UDP. I read the received counter value while in debugging mode in LabVIEW.

The issue is this: if the data wasn't immediately received on the cRIO, the data was queued and was then read eventually in the original order that it was sent. For example, the counter on the laptop would be counting through 200 and the received packet on the cRIO would be paused at 180 for a second. Then, when the cRIO started receiving packets again, it would start counting up again from 181 even though the laptop was now on 220. For a real-time system, I would expect that all missed/delayed packets would be ignored and the most recent packet would be read.

This is a real problem if I'm trying to receive target aim and distance data from the image processing on a remote computer. I have no idea how much delay there is from what I'm receiving to when it was sent. If some delay occurs, I could be off by X seconds for the remainder of the match.

I ran the same experiment with TCP and did not see this issue.

[Alan Anderson]

Chris, what caused the packets to be missed/delayed in the first place? I've been using UDP to send the vision processing results to the robot without noticeable issues. Can you show the code you used to test it?

[Chris Hibner]

Originally Posted by Alan Anderson

Chris, what caused the packets to be missed/delayed in the first place? I've been using UDP to send the vision processing results to the robot without noticeable issues. Can you show the code you used to test it?

As soon as I saw that behavior, I deleted the code and replaced it with the TCP version. I was reading that TCP has error checking for delayed and missed packets while UDP didn't, so I thought I would give TCP a try immediately.

The code was basically the example UDP client / UDP server code from the example folder, just reworked to pass the counter. On the cRIO side, the receiving was being done in the fast loop of Periodic Tasks (which we have set to 20 ms). The laptop was running a bare-bones dashboard project, and the transmit was being done in a 20 ms loop.

I'm pretty sure the problem on the receive side was we were over loading the CPU occasionally (sometimes more than occasionally). We have a lot of test code in there and running in debug mode slows things down even further. However, the behavior seen is still troublesome because I can almost guarantee there will be occasions that we'll overrun the CPU this season.

The troublesome part was that the transmission is a straight-forward FIFO queue. In the example of my previous post, if we had stopped the laptop when the counter was at 250 and the cRIO was showing the counter at 210, the cRIO would show the counter counting up 211, 212, etc as if the laptop was still alive and sending, even though it was stopped. The cRIO would continue to count up until it reached 250 (the value that the laptop was at when we stopped it), and THEN it would notice that it was stopped. The cRIO side didn't notice an issue until about 2 seconds AFTER stopping the laptop from transmitting.

[dellagd]

Yes! Careful with UDP. The difference between UDP and TCP is a pro vs con thing. UDP is faster, but you do trade of accuracy. From where im sitting (Hehe..Apollo 13), UDP just really transmits data. thats is. What happens to said data is not the senders problem. It just transmits and whatever gets there gets there. (We see this alot with the NetConsole(Which is UDP based) when some print statements will loose a letter in the beginning sometimes). With TCP, It actually, Im not sure how this part works, but it makes sure the data gets there. This process takes time, and is therefore slower.

Pro vs Con , as I said before.

[Chris Hibner]

Originally Posted by dellagd

UDP just really transmits data. thats is. What happens to said data is not the senders problem. It just transmits and whatever gets there gets there.

Yeah, I read that in the documentation. I was keeping my fingers crossed that the receiver side would be a little more like SPI or async PAS5 communication: if the receiver didn't want to use the data, it just gets overwritten by the next message. I didn't realize that when the receiver finally decided to get the message, you don't know how many messages ago you were getting. Oh well, TCP seems to be working for us. However, I may find a way to break that as well.

[Greg McKaskle]

I'm a little surprised by what you describe about the UDP test, but that is also written a bit differently that I'd normally write it. Rather than delay the UDP read loop, I'd have the loop be throttled by the UDP Read timeout parameter. As for whether the UDP buffer size, I'm sure the buffers are limited in size, but I'm not sure of the size or policy.

Greg McKaskle

[Chris Hibner]

Originally Posted by Greg McKaskle

I'm a little surprised by what you describe about the UDP test, but that is also written a bit differently that I'd normally write it. Rather than delay the UDP read loop, I'd have the loop be throttled by the UDP Read timeout parameter. As for whether the UDP buffer size, I'm sure the buffers are limited in size, but I'm not sure of the size or policy.

Greg McKaskle

Thanks for the response. I haven't tried changing the buffer or playing with the timeout. I might try that if I get some time.

[wireties]

The classic illustration if that UDP is like sending a letter. You write it, put it in an envelope and send it to the destination. It probably gets there, especially if the destination is local. BUT there is no way to be sure unless you build an acknowledgement into your application protocol.

TCP is more like a phone call. There is a distinct connection phase followed by the exchange of information (acknowledged by each party) and a disconnect phase.

UDP is faster - on highly reliable links. TCP is made to work between any two endpoints no matter the delay and reliability of the link. TCP will keep counts of packets going back and forth, ack them all and automagically retry if something is dropped. UDP just sends and forgets.

In this case I would try UDP and enumerate the packets. If you are missing a packet and/or one is out of order just throw that image away and wait for the next. Like Alan said, it should not happen very often.

And the socket address is a combination of the IP address AND the port number. So the ports to and from your robot to the driver station should always be available to you. Their availability have little to do with other robots on the field.

HTH

[cbf]

AndyB871 --

Yes, I'm very familiar with networking. (I'm a mentor, not a student.) I had assumed since the rules said port 1130 was the only one available for Operator station to Robot communication that they meant it and therefore the driver station must use that port. (I tried reading the DrvrStn COM vi, but the important stuff all seems to be buried in the opaque NETCOM library.)

Having that port available for our own purpose will make this much easier. I was getting ready to Wireshark this to see what was going on, so thanks for saving me the trouble!

Chris --

Yes, you cannot assume that subsequent UDP packets will simply replace earlier UDP packets that have not yet been read. Most operating system network stacks (and I presume that would include VxWorks) helpfully provide a certain amount of buffering to held several packets. If your packets are really short, then the network stack will hold more of them for you!

I wouldn't switch to TCP -- that just adds more overhead. The answer is either to throttle (would require back traffic), or have your code on the cRIO flush packets to catch up.

[Alan Anderson]

It's very likely that the 20 ms sampling of Chris's receiving UDP socket was at the root of the problem. If the Driver Station's 20 millisecond UDP send was running just slightly faster than the one on the cRIO -- which it almost certainly would be if the cRIO code were being run in debug mode -- then packets would start to stack up. Since the cRIO was only retrieving one packet each 20 milliseconds, it could never catch up once it got behind. If the UDP receive function itself were used to control the loop rate, with a few seconds (or more) timeout, it wouldn't fall behind the way the original code can.

[Greg McKaskle]

I think we are all getting on the same page. If you were to use WireShark, and please do if you have the time to expose the kids to it, you will see that the heartbeat of the system are UDP packets flowing from DS to robot and back every 20ms. If they glitch for more that 100ms, the robot gets disabled because the NETCOMM library doesn't feed the FPGA with the next encoded cookie.

I can't say how much UDP buffering there is, but there is definitely some. I'm sure it can be adjusted, but I'd think of that as being helpful in making the OS smaller, not more realtime. The key to having all the other loops doing UDP is to allow all of the readers to be clocked by arriving packets. Nothing else throttles them -- most have a timeout of one second in order to set some defaults. The writer loops are the ones that have the delay, and for that, I use a timed loop which is critical priority. It only takes one timed loop for the whole system and everything else responds to the sent packet. Other writing loops such as for the Kinect use regular loops.

If you want to see the packet statistics, the Charts tab on the DS now displays the lost packets per second and the trip time. Approximately 50 are sent, and the scale for total trip time to and from the robot is 100ms.

Please let me know if you see other odd behaviors or have questions.

Greg McKaskle

[Chris Hibner]

I thought about this a little more, and the behavior of the UDP seems logical to me now. The protocol is made for transferring contiguous data (like strings or files) where missing a packet is a bad idea (it would make your string nonsense or corrupt your file). Therefore, making sure all of the data is received is more important than the timing of the data.

Keep in mind that prior to this experiment, my knowledge of TCP or other comm protocols aimed at the PC world is virtually nil. My experience is more in the real-time sensor protocols where the data is usually single sensor readings that change rapidly. In my world, receiving everything that is sent is usually much less important than making sure you have the most recent data (although in an ideal world, you like to have both).

I guess we're kind of trying to use a square peg to fit through a round hole. I'm seeing that you may have to whittle the peg down a bit, but eventually you can get it to fit. I think I have what I need to make it work.

[Joe Ross]

Originally Posted by Chris Hibner

I thought about this a little more, and the behavior of the UDP seems logical to me now. The protocol is made for transferring contiguous data (like strings or files) where missing a packet is a bad idea (it would make your string nonsense or corrupt your file). Therefore, making sure all of the data is received is more important than the timing of the data.

Both TCP and UDP are typically implemented with multiple levels of fifos (buffer in the library at the send side, buffer in the os in the send side, buffer in the ethernet chip in the send side, buffer in the ethernet chip at the receive side, etc). This does complicate only reading the most recent data. I have not looked at the LabVIEW UDP libraries, but (based on experience with UDP on other embedded systems) I would expect there is a way to peek into the buffer to see how much is waiting, read all of it, then only use the latest data in your code.

If you research UDP vs TCP, you'll actually find the opposite conclusion (UDP loses data, TCP doesn't). However, this comes from use over a lossy connection, such as the internet. UDP just blasts out the data. If one packet gets routed through Russia, it will be received after a later packet that gets routed more directly. It can also get lost. It's used for protocols where missing or out of order information is not critical, but low latency is. TCP on the other hand is a handshaked protocol where if a packet is lost, it will be re-transmitted or if a packet is delayed, it will be re-assembled in the correct order. This is what you use if want to transfer contiguous data. Neither tries to lose data, but only TCP guarantees you don't lose data.

Now, in a low utilization local link like the robot network, in reality packets don't get lost or delayed, so there is little difference between TCP and UDP.

I'm not sure why you see different results with tcp. If anything, the results should be worse, given the same algorithm for reading data.

[Chris Hibner]

Originally Posted by Joe Ross

I'm not sure why you see different results with tcp. If anything, the results should be worse, given the same algorithm for reading data.

I'm thinking I would attribute that to sample size (i.e. I didn't do enough testing to see it yet).

[Jon236]

Originally Posted by Greg McKaskle

I'm a little surprised by what you describe about the UDP test, but that is also written a bit differently that I'd normally write it. Rather than delay the UDP read loop, I'd have the loop be throttled by the UDP Read timeout parameter. As for whether the UDP buffer size, I'm sure the buffers are limited in size, but I'm not sure of the size or policy.

Greg McKaskle

That worked for us.