Packet Loss Spikes at 1 Minute Intervals

[jspatz1]

Like everyone else we are working to understand and minimize the potential for field communication troubles heading into Champs. As we watch the data packet charts we are noticing a peculiar pattern: Brief spikes of packet loss at exactly 1 minute intervals. The pattern is so repeatable our co-pilot can predict when it will occur by watching the chart, and the driver then confirms on cue a momentary loss of control. This jives with the driver's reports from some competition matches, when he briefly lost control "once or twice per match."

We use Labview. We have seen the same pattern while driving with two different computers, a Classmate and an Asus i5 laptop, both Windows 7.

Wondering if anyone else has seen this phenomenon, know its cause, or cares to speculate on the source of such a pattern. Our only guess is Windows activity.

[iyermihir]

I would recommend looking at the cpu usage on the cRIO. It is displayed in the charts section of the driver station. We had a problem where our usage was at 100% and the robot would not respond immediately. We fixed the problem by adding a 1ms sleep in both our main and vision loops. I am not sure if this is the cause of your problem, but its worth a look. By the way, we use Java.

[Greg McKaskle]

I think your guess about Win7 activity is a good one. I've seen spikes of about 70 seconds on some other teams. If the spikes are there in all robot modes, even disabled, and there when tethered, that is a strong indicator that it is the laptop SW.

If you open the task manager, sort by CPU usage, and wait for the spike, you should see a process jump to the top. I have seen web page content cause it, and be warned that task manager will make its own one second ripple. Once you know what it is, you should be able to take appropriate action.

I'm curious if you are running the Windows Explorer, of if you setup the DS as the shell? Another thing you could try is to elevate the priority of the DS, but don't get carried away.

Greg McKaskle

[Bill_B]

You can choose the task manager's refresh cycle. Try several values to see the effect of task manager refresh on your observations.

There is also the button on the processes' tab screen that lets you see process from all users. Click it to include system processes in the display.

[jspatz1]

Originally Posted by Greg McKaskle

I'm curious if you are running the Windows Explorer, of if you setup the DS as the shell? Another thing you could try is to elevate the priority of the DS, but don't get carried away.

Greg McKaskle

Greg,

Would appreciate if you could send us some instruction on how to do this.

[404'd]

Originally Posted by jspatz1

Greg,

Would appreciate if you could send us some instruction on how to do this.

You can elevate the priority of a process using Task Manager. "Don't get carried away" meaning, setting priority to realtime can cause everything else not to respond.

EDIT: The picture shows elevating another process, obviously you'd elevate the driver station.exe process

[Radical Pi]

Originally Posted by jspatz1

Greg,

Would appreciate if you could send us some instruction on how to do this.

Also, if you mean on how to make the DS run without explorer (trust me, it helps), here are the instructions I use when setting up a new computer.

First of all, log into the Driver account. Optionally, change the background to a solid dark grey, which imo looks fairly nice with the driver station

There are two different ways you can go from here, depending on which version of windows 7 you have. The second one will work on any version.

For Professional or better: Start->Run gpedit.msc. Browse to User Configuration->System->Custom User Interface, and set it to “C:\Program Files\FRC Driver Station\Driver Station.exe”, without quotes, or “C:\Program Files (x86)\FRC Driver Station\Driver Station.exe” if you have 64-bit windows. Log out and back in, and you should have only the Driver Station.

For all other versions: Start->Run regedit.exe. Be very careful here, all the usual warnings about registry editing apply. Browse to HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon. Create a new String value named Shell, and set it to “C:\Program Files\FRC Driver Station\Driver Station.exe”, without quotes, or “C:\Program Files (x86)\FRC Driver Station\Driver Station.exe” if you have 64-bit windows. Log out and back in, and you should have only the Driver Station.

If you ever need it, you can manually launch explorer by opening Task Manager (Ctrl-Alt-Del), hitting "New Task...", and entering explorer.exe.

Good Luck, and see you at St. Louis!

[jspatz1]

Ian,

Thanks for helping with the info. We have tried the regedit you described and are not getting the DS to come up as the shell. Have checked it pretty carefully . A screen shot of the regedit window is attached. Windows 7 Home Premium 64-bit.

There is a Shell folder under the Windows folder, anything need to be done there?

[Joe Ross]

Did you ever get to the bottom of the packet loss spikes at 1 minute intervals? I've seen it on a few computers now.

[Jared]

We had this exact issue, a second of lost communication every 60 seconds, which was fixed by doing one of these things.

- Remove printer driver and network printer service

- Unjoin domain, update network interface drivers

- Update display driver and many others

- Eliminate all unnecessary services and startup programs

- Uninstall USB firewire, eSata, and CD drive drivers

[Greg McKaskle]

An odd question, but do you think you can cause it to happen again? Please?

If you have a machine and driver combo, I'd like to see if process elevation or other things can fix this. The UDP send loop is already elevated, but I don't have much to stress test it.

Greg McKaskle

[G_rupp]

Greg, this happens in our shop on multiple computers when running wireless. Our Drive Station computer is an XP computer with almost everything disabled and it does not have any virus scanning software. We have used both 2.4GHz and 5GHz and still get the disconnect about every minute.

I believe that it will also happen on my Windows 7 computer. I will run it tonight and verify. Are there any other items that I could try for you?

I may be able to send an XP Drive Station computer to you.

There are 2 other threads on this. http://www.chiefdelphi.com/forums/sh...d.php?t=127469
http://www.chiefdelphi.com/forums/sh...d.php?t=126946

[Greg McKaskle]

Does the DS disconnect or show a spike of lost packets? If possible can you attach a picture of your DS log file that shows one of these symptoms?

Greg McKaskle

[G_rupp]

I have attached the image and the dslog and dsevents from the XP drive station.

[Greg McKaskle]

Several people have provided log files. Thank you.

One log file showed spikes every 69 seconds that lasted 5 or 6 seconds.

The other showed spikes every 126 seconds that last for 2 seconds. One interval was 189 seconds instead. That implies a potential base period of 63 seconds.

Thus far, I haven't found any Windows service that people complain about with either of those periods, and I don't have a computer that shows the problem.

So, if you have one that is misbehaving, my suggestion is to go to the control panel and search for Schedule. Open the Task scheduler and look at the list of tasks that ran in the last 24 hours. Toggle them open to see how often, how long, and their completion status.

You may also want to click on the local item definitions in the leftmost tab. This shows how they are defined and allows you to change them. Any task that is set to run with elevated privileges may be capable of this interruption.

It may also be useful to look at the services. Search for Service in the Control Panel. There is less information about when the services actually do something, but you may find some things that raise a flag.

I'd then probably rank these and turn off those most likely to cause issues. Please, please, post the results.

If I can get this to happen on a computer, I'll poke around as well.

Greg McKaskle