Chief Delphi - What does the Camera Slow Down?

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Programming (http://www.chiefdelphi.com/forums/forumdisplay.php?f=51)

- - What does the Camera Slow Down? (http://www.chiefdelphi.com/forums/showthread.php?t=84611)

What does the Camera Slow Down?

I've heard from a few people now that the camera slows down the dashboard. However, I'm not exactly sure what, how, or why it slows down. So...

If the dashboard slows down, is it only the dashboard? (For instance, the dashboard does update as frequently, but the robot still responds the same)

If the robot slows down, why and how? Is there a way around this?

Is the problem because people are sending the image to the dashboard? What if we don't do this?

I think you can get the general idea of what I'm looking for. We're trying to decide if the camera is worth the time/effort.

Re: What does the Camera Slow Down?

A few things
Generally, things slow down because there is either a lack of RAM, CPU power, or in some cases bandwidth.

1. The camera has to send complete images over WiFi 15 times per second. That's a lot of bandwidth, something WiFi can't always handle as fast as we would like.

2. The Dashboard Data is updated at the same frequency as the robot comm packets, 20hz or every 50ms. The Camera image is different, updating as fast as it can (which isn't very fast, usually around 10hz max)

3. If the robot slows down, either you are sending too much data and the field is throttling bandwidth, or your code is using more than the CPU can provide. I have seen both. If you only have a problem on the field, it's the first. Otherwise, it is probably the second. The solution is to compress the image more and send less (smaller image), or lessen use on CPU.

4. We have had problems with the Dashboard image, but we didn't use it anyway so we disabled it.

Re: What does the Camera Slow Down?

Are there any suggestions for what you can do in code (we use Java) to make image processing more efficient so that it doesn't affect robot performance?

Re: What does the Camera Slow Down?

Quote:

Originally Posted by Geek 2.0 (Post 942456)

Are there any suggestions for what you can do in code (we use Java) to make image processing more efficient so that it doesn't affect robot performance?

We use this and haven't noticed any slow down:

Code:

cam = AxisCamera.getInstance();

cam.writeResolution(AxisCamera.ResolutionT.k320x240)

cam.writeCompression(30);

The image size/resolution and compression are the things to pay attention to as obviously the less compressed and larger the picture, the more bandwidth would be used up.

Re: What does the Camera Slow Down?

Is there any way to disable the sending of the video over the wire to the dashboard? We used the stock dashboard and only enabled the camera (at 320x240), and found that our controls were ~1 second delayed when we got on the field at portland.

We ended up just disabling all our camera code and just leaving the initialization and still saw a huge slow down, so we just decided to just give up ont eh camera entirely.

Re: What does the Camera Slow Down?

Quote:

Originally Posted by ErichKeane (Post 942500)

When you instantiate the AxisCamera object, it creates the video server task behind the scenes. If you comment out the instantiation the server won't be started and video won't be sent.

EDIT: After rereading your post it sounds like you might be trying to use the camera onboard without sending video back. If that's the case you would need to specifically update the WPILib code to not start the server automatically.

Re: What does the Camera Slow Down?

Quote:

Originally Posted by Dave Scheck (Post 942506)

When you instantiate the AxisCamera object, it creates the video server task behind the scenes. If you comment out the instantiation the server won't be started and video won't be sent.

I guess my point is: I want to be able to use the data from the camera on the cRio without streaming it to the Dashboard, which I suspect is the reason for the huge delay we saw when we got on the field.

Re: What does the Camera Slow Down?

Don't know if you saw this yet:

Quote:

Originally Posted by Dave Scheck (Post 942506)

EDIT: After rereading your post it sounds like you might be trying to use the camera onboard without sending video back. If that's the case you would need to specifically update the WPILib code to not start the server automatically.

Re: What does the Camera Slow Down?

Quote:

Originally Posted by synth3tk (Post 942537)

Don't know if you saw this yet:

I didn't, thanks! He edited after I replied and was done. I was afraid of that answer... At least it'll give me something to do next year :)

Re: What does the Camera Slow Down?

You really should be able to get it to work fine. Take a look at these threads:

http://www.chiefdelphi.com/forums/sh...ht=java+camera

But, seriously, if you haven't tried compressing the image, you will find that it will solve most lag issues (unless there are problems elsewhere in your code).

Re: What does the Camera Slow Down?

Interesting points in that Tuba, thanks.

I doubt it was the rest of the code, it was extremely simple code, particularly since the camera stuff wasn't there. Basically we got to Portland for a practice round, found it was really slow, so we disabled OUR camera code (left the initialization) and it was still really slow. So we removed the initialization and everything was fine.

We had compression at 80 (i think that was the recommended number last year?).

We were fine with the rest of the stuff on that list (other than resolution), since we didn't ever even use the developer account.

It worked perfectly either wired or on the practice field, but once we got to the actual field, there was a huge delay according to the driver.

Re: What does the Camera Slow Down?

Quote:

Originally Posted by apalrd (Post 942455)

1. The camera has to send complete images over WiFi 15 times per second. That's a lot of bandwidth, something WiFi can't always handle as fast as we would like.

2. The Dashboard Data is updated at the same frequency as the robot comm packets, 20hz or every 50ms. The Camera image is different, updating as fast as it can (which isn't very fast, usually around 10hz max)

3. If the robot slows down, either you are sending too much data and the field is throttling bandwidth, or your code is using more than the CPU can provide. I have seen both. If you only have a problem on the field, it's the first. Otherwise, it is probably the second. The solution is to compress the image more and send less (smaller image), or lessen use on CPU.

4. We have had problems with the Dashboard image, but we didn't use it anyway so we disabled it.

There's more than enough bandwidth to send the video. The main reason for the slowdown is how data is sent, not how much.

Networking 101:

In the modern world of networking, there are two main network protocols, UDP and TCP.

TCP (Transmission Control Protocol) comes first. It is an extremely common and robust protocol (http, the "internet protocol", runs on top of it). It works by forming a connection between two devices and then sending data across the connection. The protocol is managed: code running behind the scenes makes sure that the data arrives to the program using a TCP socket undamaged and in order. It does this in the following ways:

If a packet doesn't make it across the network intact, TCP stops all other incoming packets from being received and requests the bad packet to be resent.
If a packet arrives out of order, the TCP socket waits to deliver it until the packets that come before it in the stream arrive.

This can make TCP bad for use in real-time applications where getting new data (such as frames in a video stream) is more important than waiting for the old data to come in undamaged.

UDP (User Datagram Protocol) is the second type, and the simpler of the two. It is a connectionless protocol. Unlike TCP, sending data over a UDP socket just "shouts" it at the target IP without first forming a connection. There is no confirmation that the packets are received. The header of each packet sent across the network contains only its source, its destination, a packet length, and a checksum to confirm that this data hasn't been mangled. Because of its connectionless nature and small header, UDP provides no guarantee that the packets arrive intact or in order. It is up to the programmer to assure these things using data they send in the packets.

However, the less-structured nature of UDP can be an advantage in real-time applications where getting new data (such as frames in a video stream) is more important than waiting for old data to be resent or be put into order. Video chat programs such as Skype usually use UDP because of this.

How it relates:

The robot comm packets and dashboard packets are sent in UDP, which makes sense since up-to-date information is the critical factor here. However, for a reason that I cannot understand in any way, the video is sent using TCP. Consider why this is bad:

If a frame of video is bad, it would make the most sense for the dashboard program to just skip it and move onto the next frame. However, this is impossible in TCP, as the connection itself stops until it can retrieve the bad frame, slowing down the dashboard.
Not only is the dashboard slowed down, but the robot is slowed down too. The networking task in charge of sending the video has to stop sending new information in order to resend the bad data. This can cause performance issues in the robot as more time is spent "catching up" and less time is spent executing other code.

TheDominis and I appealed to the GDC some time ago to be allowed to develop our own UDP video system, but for whatever reason we were shut down. I'm still at a loss for why, but for now we're stuck with the TCP video feed.

Re: What does the Camera Slow Down?

Useful post for many of the students here I'd suspect. All you caused for me is flashbacks of the 7 layer OSI model which I learned about for years. I'm intimately familiar with TCP vs UDP :)

I'd realized that the video here was sent via TCP (fired up Ethereal while dianosing issues during the build season).

What I figured was that there wasn't enough bandwidth for 6 robots running the video and other traffic at the competition, which is why it runs fine on the practice field/elsewhere, but not on the competition field.

I was really hoping there was an switch I could set in the initialization code that would disable the server, so I would be able to do my normal analyzation, and not the dashboard view (since I knew about the no UDP stuff form the GDC).

Re: What does the Camera Slow Down?

The problem seems to have much more to do with the Classmate having issues rendering video and working the TCP stream at the same time due to its absolutely [sarcasm]wonderful processor[/sarcasm]. It seems that the best/only solution at present time is to compile WPILib yourself and not initialize the PCVideoServer when the camera is initialized.

Re: What does the Camera Slow Down?

Quote:

Originally Posted by slavik262 (Post 942668)

It seems that the best/only solution at present time is to compile WPILib yourself and not initialize the PCVideoServer when the camera is initialized.

Yep, seems like that is the case. Perhaps it would give me a reason to fix the WPILib like I've been promising myself for years :)

Re: What does the Camera Slow Down?

If you want to lower the overhead of the Dashboard, open the source project and right click on the image display and toggle Visible>>Image Information to be off. You can achieve similar effect by deleting the chart or modifying the chart drawing region so that it is not overlapped by the Y scales.

The combination of these two drawing elements that invalidate in order to update, will cause a large section of the screen to be redrawn, not just the modified areas.

I don't know enough about the Java video to guarantee that this will get rid of the lag, but it will not hurt either.

As for using raw TCP to transfer video, the underlying camera uses ...
wait for it
...
wait for it
...
TCP.

Sorry, but I couldn't resist. My two year old now says that -- the wait for it part.

While I agree that it is not the ideal protocol, it isn't as bad as described, because it is the underlying IP packets that are acked and retransmitted, not the entire TCP message. Since it is not possible to decode a jpeg past a missing packet, incomplete jpegs do need to be detected and dealt with. Ideally, the images would be sent over RTP or a similar specialization of UDP.

Actually, as the topology of the robot network changes, the camera may someday be moved to be on a common switch so that the camera can serve up images to both the robot and DB, meaning that no image server is needed. That'd be nice.

Greg McKaskle

Re: What does the Camera Slow Down?

Quote:

Originally Posted by ErichKeane (Post 942670)

Yep, seems like that is the case. Perhaps it would give me a reason to fix the WPILib like I've been promising myself for years :)

For any improvements you make, please submit patches as artifacts on the wpilib project at firstforge.wpi.edu

Thanks,
-Joe

Re: What does the Camera Slow Down?

Quote:

Originally Posted by slavik262 (Post 942668)

The problem seems to have much more to do with the Classmate having issues rendering video and working the TCP stream at the same time due to its absolutely [sarcasm]wonderful processor[/sarcasm].

Is there absolutely nothing to be done about gutting the WinXP install on the Classmates to be able to do more with that chip? I'm aware that hardware-wise, we're screwed, but are we allowed to disable background processes that the Classmates won't be needing for our purposes (such as the Print Spooler)? Someone pointed me at one of the rules in the rulebook, but my machine here at home won't download it properly...

Re: What does the Camera Slow Down?

While it may be cathartic to gut something every now and then, you may want to look at the task manager before spending the effort. I think you'll find that while there are threads and processes spun up, they aren't doing much.

You will get much better results by hiding the label I mentioned. In hindsight, I'd have hidden it myself, but I thought the performance was acceptable.

Also, I found this info using the profiler built into LV. If you are trying to speed something up, I highly recommend taking measurements first, to know what is slow.

If you have other questions, please post.
Greg McKaskle

Re: What does the Camera Slow Down?

Quote:

Originally Posted by Greg McKaskle (Post 943368)

Also, I found this info using the profiler built into LV

I don't have LV in front of me, so forgive me if this is something that can be figured out in a few seconds of playing around. Where is the profiler located? Is there anything that would be beneficial to know for a first time user? I'm assuming this should be run on the classmate to get the most bang for the buck and I'm also assuming that it has to be run in the developer account (with no way to profile in the driver account). Thanks!

Re: What does the Camera Slow Down?

I'll be happy to give a bit of guidance.

As with most profilers, it isn't hard, but getting good data and finding the needle in the haystack sometimes takes some finesse.

You will want to run in the developer account, or possibly run on a different computer -- its what I did. By the way, once you use it on the dashboard, you may find it useful on the cRIO too, if using LV.

The profiler is located under Tools>>Profile>>Performance and Memory. It opens the profile window. Open the project for the dashboard, open the dashboard panel, its diagram, start the driver station, killing the DB it opens if necessary. Open the profiler and ...

1. Press Start on the profile window.
2. Run the dashboard for five or ten seconds.
3. Abort the dashboard.
4. Press Stop on the profile window.

This should give you a table. Rows of VIs with columns for VI time, subVI time, and total time. To hunt for the hot spot, it is often useful to sort by different columns by clicking on the header above the column. Clicking on a row hilights it to more easily follow it and absorb what the row is telling you. Additionally, it is often useful to double click a VI row to see how that item was affected by subVI calls. This lets you see how common subVI calls are charged to different callers. You can right click on a row to access a contextual menu that aids navigation to hierarchy, to the VI, or to callers. You may also find it useful to turn on Timing Statistics (after the fact is fine). I don't use it for statistics so much as number of calls to spot n-squared algorithms, etc. Also useful (after the fact is fine) is the Timing Details to see drawing time, display time, etc.

Saving the profile data will put it into a spreadsheet file. You cannot load it into the profile window again, but you can make due with loading several sheets into Excel and doing your own before and after comparisons.

The profiler works at the VI level, meaning it is a little coarse, but since it is pretty easy to make subVI from selection, it is usually good enough to get you in the ballpark of the hotspot, and if not, you can introduce new profile VIs easily. Similarly, you can comment out code using the Diagram Disable structure to surround code.

Note:
The profiler only records and reports complete VI runs. If a VI was already running when the profiler was started, its time will not be in the report. For the complete picture, start the profile before you profile what you care about. If you know that you are calling into the subVIs where the hotspot likely is, you can Start mid-application and hit the snapshot button to see intermediates. This is very useful for ad hoc measurements, but be careful to remember what you aren't profiling by doing this.

Let me know what else is confusing. Hope this helps.
Greg McKaskle