What does the Camera Slow Down?

[Geek 2.0]

I've heard from a few people now that the camera slows down the dashboard. However, I'm not exactly sure what, how, or why it slows down. So...

If the dashboard slows down, is it only the dashboard? (For instance, the dashboard does update as frequently, but the robot still responds the same)

If the robot slows down, why and how? Is there a way around this?

Is the problem because people are sending the image to the dashboard? What if we don't do this?

I think you can get the general idea of what I'm looking for. We're trying to decide if the camera is worth the time/effort.

[apalrd]

A few things
Generally, things slow down because there is either a lack of RAM, CPU power, or in some cases bandwidth.

1. The camera has to send complete images over WiFi 15 times per second. That's a lot of bandwidth, something WiFi can't always handle as fast as we would like.

2. The Dashboard Data is updated at the same frequency as the robot comm packets, 20hz or every 50ms. The Camera image is different, updating as fast as it can (which isn't very fast, usually around 10hz max)

3. If the robot slows down, either you are sending too much data and the field is throttling bandwidth, or your code is using more than the CPU can provide. I have seen both. If you only have a problem on the field, it's the first. Otherwise, it is probably the second. The solution is to compress the image more and send less (smaller image), or lessen use on CPU.

4. We have had problems with the Dashboard image, but we didn't use it anyway so we disabled it.

[Geek 2.0]

Are there any suggestions for what you can do in code (we use Java) to make image processing more efficient so that it doesn't affect robot performance?

[TubaMorg]

Originally Posted by Geek 2.0

Are there any suggestions for what you can do in code (we use Java) to make image processing more efficient so that it doesn't affect robot performance?

We use this and haven't noticed any slow down:

cam = AxisCamera.getInstance();
cam.writeResolution(AxisCamera.ResolutionT.k320x240)
cam.writeCompression(30);

The image size/resolution and compression are the things to pay attention to as obviously the less compressed and larger the picture, the more bandwidth would be used up.

[ErichKeane]

Is there any way to disable the sending of the video over the wire to the dashboard? We used the stock dashboard and only enabled the camera (at 320x240), and found that our controls were ~1 second delayed when we got on the field at portland.

We ended up just disabling all our camera code and just leaving the initialization and still saw a huge slow down, so we just decided to just give up ont eh camera entirely.

[Dave Scheck]

Originally Posted by ErichKeane

Is there any way to disable the sending of the video over the wire to the dashboard? We used the stock dashboard and only enabled the camera (at 320x240), and found that our controls were ~1 second delayed when we got on the field at portland.

We ended up just disabling all our camera code and just leaving the initialization and still saw a huge slow down, so we just decided to just give up ont eh camera entirely.

When you instantiate the AxisCamera object, it creates the video server task behind the scenes. If you comment out the instantiation the server won't be started and video won't be sent.

EDIT: After rereading your post it sounds like you might be trying to use the camera onboard without sending video back. If that's the case you would need to specifically update the WPILib code to not start the server automatically.

[ErichKeane]

Originally Posted by Dave Scheck

When you instantiate the AxisCamera object, it creates the video server task behind the scenes. If you comment out the instantiation the server won't be started and video won't be sent.

I guess my point is: I want to be able to use the data from the camera on the cRio without streaming it to the Dashboard, which I suspect is the reason for the huge delay we saw when we got on the field.

[synth3tk]

Don't know if you saw this yet:

Originally Posted by Dave Scheck

EDIT: After rereading your post it sounds like you might be trying to use the camera onboard without sending video back. If that's the case you would need to specifically update the WPILib code to not start the server automatically.

[ErichKeane]

Originally Posted by synth3tk

Don't know if you saw this yet:

I didn't, thanks! He edited after I replied and was done. I was afraid of that answer... At least it'll give me something to do next year

[TubaMorg]

You really should be able to get it to work fine. Take a look at these threads:

http://www.chiefdelphi.com/forums/sh...ht=java+camera

But, seriously, if you haven't tried compressing the image, you will find that it will solve most lag issues (unless there are problems elsewhere in your code).

[ErichKeane]

Interesting points in that Tuba, thanks.

I doubt it was the rest of the code, it was extremely simple code, particularly since the camera stuff wasn't there. Basically we got to Portland for a practice round, found it was really slow, so we disabled OUR camera code (left the initialization) and it was still really slow. So we removed the initialization and everything was fine.

We had compression at 80 (i think that was the recommended number last year?).

We were fine with the rest of the stuff on that list (other than resolution), since we didn't ever even use the developer account.

It worked perfectly either wired or on the practice field, but once we got to the actual field, there was a huge delay according to the driver.

[slavik262]

Originally Posted by apalrd

1. The camera has to send complete images over WiFi 15 times per second. That's a lot of bandwidth, something WiFi can't always handle as fast as we would like.

2. The Dashboard Data is updated at the same frequency as the robot comm packets, 20hz or every 50ms. The Camera image is different, updating as fast as it can (which isn't very fast, usually around 10hz max)

3. If the robot slows down, either you are sending too much data and the field is throttling bandwidth, or your code is using more than the CPU can provide. I have seen both. If you only have a problem on the field, it's the first. Otherwise, it is probably the second. The solution is to compress the image more and send less (smaller image), or lessen use on CPU.

4. We have had problems with the Dashboard image, but we didn't use it anyway so we disabled it.

There's more than enough bandwidth to send the video. The main reason for the slowdown is how data is sent, not how much.

Networking 101:

In the modern world of networking, there are two main network protocols, UDP and TCP.

TCP (Transmission Control Protocol) comes first. It is an extremely common and robust protocol (http, the "internet protocol", runs on top of it). It works by forming a connection between two devices and then sending data across the connection. The protocol is managed: code running behind the scenes makes sure that the data arrives to the program using a TCP socket undamaged and in order. It does this in the following ways:

• If a packet doesn't make it across the network intact, TCP stops all other incoming packets from being received and requests the bad packet to be resent.
• If a packet arrives out of order, the TCP socket waits to deliver it until the packets that come before it in the stream arrive.

This can make TCP bad for use in real-time applications where getting new data (such as frames in a video stream) is more important than waiting for the old data to come in undamaged.

UDP (User Datagram Protocol) is the second type, and the simpler of the two. It is a connectionless protocol. Unlike TCP, sending data over a UDP socket just "shouts" it at the target IP without first forming a connection. There is no confirmation that the packets are received. The header of each packet sent across the network contains only its source, its destination, a packet length, and a checksum to confirm that this data hasn't been mangled. Because of its connectionless nature and small header, UDP provides no guarantee that the packets arrive intact or in order. It is up to the programmer to assure these things using data they send in the packets.

However, the less-structured nature of UDP can be an advantage in real-time applications where getting new data (such as frames in a video stream) is more important than waiting for old data to be resent or be put into order. Video chat programs such as Skype usually use UDP because of this.

How it relates:

The robot comm packets and dashboard packets are sent in UDP, which makes sense since up-to-date information is the critical factor here. However, for a reason that I cannot understand in any way, the video is sent using TCP. Consider why this is bad:

1. If a frame of video is bad, it would make the most sense for the dashboard program to just skip it and move onto the next frame. However, this is impossible in TCP, as the connection itself stops until it can retrieve the bad frame, slowing down the dashboard.
2. Not only is the dashboard slowed down, but the robot is slowed down too. The networking task in charge of sending the video has to stop sending new information in order to resend the bad data. This can cause performance issues in the robot as more time is spent "catching up" and less time is spent executing other code.

TheDominis and I appealed to the GDC some time ago to be allowed to develop our own UDP video system, but for whatever reason we were shut down. I'm still at a loss for why, but for now we're stuck with the TCP video feed.

[ErichKeane]

Useful post for many of the students here I'd suspect. All you caused for me is flashbacks of the 7 layer OSI model which I learned about for years. I'm intimately familiar with TCP vs UDP

I'd realized that the video here was sent via TCP (fired up Ethereal while dianosing issues during the build season).

What I figured was that there wasn't enough bandwidth for 6 robots running the video and other traffic at the competition, which is why it runs fine on the practice field/elsewhere, but not on the competition field.

I was really hoping there was an switch I could set in the initialization code that would disable the server, so I would be able to do my normal analyzation, and not the dashboard view (since I knew about the no UDP stuff form the GDC).

[slavik262]

The problem seems to have much more to do with the Classmate having issues rendering video and working the TCP stream at the same time due to its absolutely [sarcasm]wonderful processor[/sarcasm]. It seems that the best/only solution at present time is to compile WPILib yourself and not initialize the PCVideoServer when the camera is initialized.

[ErichKeane]

Originally Posted by slavik262

It seems that the best/only solution at present time is to compile WPILib yourself and not initialize the PCVideoServer when the camera is initialized.

Yep, seems like that is the case. Perhaps it would give me a reason to fix the WPILib like I've been promising myself for years