SmartDashboard Crashing

[eddie12390]

I've just recently started using the SmartDashboard and I'm having trouble with it crashing. It'll literally only stay open for 1-1.5 minutes and then it'll just suddenly close. The major problem with this is that we use custom settings that aren't being saved when we save our SmartDashboard .xml profile. (Window size, IP of the cameras, label backgrounds)

I've tried running the jar from a Command Prompt and am not seeing any unusual output or errors.

[FourPenguins]

This may be related to http://firstforge.wpi.edu/sf/go/artf1445?nav=1&_pagenum=1&returnUrlKey=1328405797848 if you're running the camera display.

[basicxman]

This may be related to http://firstforge.wpi.edu/sf/go/artf1445?nav=1&_pagenum=1&returnUrlKey=1328405797848 if you're running the camera display.

Also possibly related: http://www.chiefdelphi.com/forums/showthread.php?t=101552

[pblankenbaker]

You might want to try this post as well:

http://www.chiefdelphi.com/forums/showpost.php?p=1120367&postcount=6

It has a zip file containing a replacement WPIJavaCV.jar file (built from source on 2012-Feb-05) that seems to have addressed the memory leak issue that we ran across while using the SmartDashboard for image processing.

NOTE: This isn't a official release of the JAR file, I just downloaded the source directly from the WPI subversion server and built it. So, if you are not in a hurry, it might be wise to wait for a official release.

[misko]

You might want to try this post as well:

http://www.chiefdelphi.com/forums/showpost.php?p=1120367&postcount=6

It has a zip file containing a replacement WPIJavaCV.jar file (built from source on 2012-Feb-05) that seems to have addressed the memory leak issue that we ran across while using the SmartDashboard for image processing.

NOTE: This isn't a official release of the JAR file, I just downloaded the source directly from the WPI subversion server and built it. So, if you are not in a hurry, it might be wise to wait for a official release.

Appears this fixes a memory leak in the contour code. But if you are running into the problem FourPenguins refers to, Artifact artf1445 : Camera causes VM crash, this new jar file doesn't fix that problem. And that problem is quite frustrating and makes using SmartDashboard impossible.

[misko]

I've just recently started using the SmartDashboard and I'm having trouble with it crashing. It'll literally only stay open for 1-1.5 minutes and then it'll just suddenly close. The major problem with this is that we use custom settings that aren't being saved when we save our SmartDashboard .xml profile. (Window size, IP of the cameras, label backgrounds)

I've tried running the jar from a Command Prompt and am not seeing any unusual output or errors.

Assuming you are using Sun/Oracle Java, open a command prompt window, add the path to java to your path, something like PATH=%PATH%;C:\Program Files (x86)\Java\jre6\bin. Then cd to where you have SmartDashboard installed, and run it as: java -jar SmartDashboard.jar

Then when it exits you will likely get more information. The problem we are seeing is with an access violation. I suspect the memory leak problem exhibited something different. You can also do java -jar -ea SmartDashboard.jar and get a little more output.

[patrickviolette]

You might want to try this post as well:

http://www.chiefdelphi.com/forums/showpost.php?p=1120367&postcount=6
I just downloaded the source directly from the WPI subversion server

How exactly did you do that? I am here: http://firstforge.wpi.edu/sf/scm/do/viewRepository/projects.smartdashboard/scm.SmartDashboard and when I try to do a checkout, it asks for authentication for the user "nobody." What is the password that you used that worked?

[misko]

How exactly did you do that? I am here: http://firstforge.wpi.edu/sf/scm/do/viewRepository/projects.smartdashboard/scm.SmartDashboard and when I try to do a checkout, it asks for authentication for the user "nobody." What is the password that you used that worked?

You need to create an account at firstforge and then checkout with that account.
http://firstforge.wpi.edu/sf/sfmain/do/createUser

[eddie12390]

I tried all of the fixes available in this thread and they just seemed to exacerbate the problem.

[misko]

I tried all of the fixes available in this thread and they just seemed to exacerbate the problem.

I'm curious, were you able to try post #6 and see the output?
We reliably get an EXCEPTION_ACCESS_VIOLATION if we run smartdashboard directly. Now if I could only figure out how to fix it...

[JDNovak]

We also are seeing the same EXCEPTION_ACCESS_VIOLATION crash after about 10 min under the same circumstances. Happening reliably on several different machines now. We're not having much luck fixing it so far.

[Mr. Lim]

Still having the same issues here. Happens on every computer we try regardless of OS or configuration. The faster the computer, the less time until we crash. The camera feed also temporarily drops for a few seconds just before the entire app crashes.

[Jared Russell]

We are also still having this issue, although it takes a good 20 minutes before it happens on our machine. I have confirmed that there are no (egregious) memory leaks remaining, but every now and then I turn around in the pit and SmartDashboard has crashed.

I have instructed our drivers to restart SmartDashboard immediately before any match, and as a result we haven't seen any problems behind the glass.

[misko]

We have one laptop it takes 5 or 10 minutes to crash, another which reliably crashes in 30 seconds. I'm going to try to use the latest OpenCV, JavaCV, and ffmpeg and see if it makes a difference. Just changing the libraries doesn't seem sufficient, appears I need to rebuild WPIJavaCV or WPICameraExtension or something to get it to use the newer libraries.

[Jefferson]

We have one laptop it takes 5 or 10 minutes to crash, another which reliably crashes in 30 seconds. I'm going to try to use the latest OpenCV, JavaCV, and ffmpeg and see if it makes a difference. Just changing the libraries doesn't seem sufficient, appears I need to rebuild WPIJavaCV or WPICameraExtension or something to get it to use the newer libraries.

We tried rebuilding WPIJavaCV tonight with the latest JavaCV libraries and got the same results. Maybe you'll have better luck. Let us know how it turns out.

[Brian Selle]

Love the SmartDashboard but we are having the same issue. We don't do any image processing using the JavaCV libraries, just display the camera feed. The JVM dump file(hs_err_pidxxxxx.log) in program files/SmartDashboard looks like:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x026f86d0, pid=13336, tid=13392
#
# JRE version: 6.0_30-b12
# Java VM: Java HotSpot(TM) Client VM (20.5-b03 mixed mode, sharing windows-x86 )
# Problematic frame:
# J java.nio.DirectByteBuffer.get()B
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x04deac00): JavaThread "Camera Background" [_thread_in_Java, id=13392, stack(0x06e60000,0x06eb0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x10e41cd5

Registers:
EAX=0x00001ca5, EBX=0x00000036, ECX=0x245d1968, EDX=0x00000000
ESP=0x06eaf730, EBP=0x06eafb4c, ESI=0x10e40030, EDI=0x00001ca4
EIP=0x026f86d0, EFLAGS=0x00010207

Top of Stack: (sp=0x06eaf730)
0x06eaf730: 06eaf754 6d88ea06 04dead28 04d52134
0x06eaf740: 04d51fc8 10e40030 00000000 000e1000
0x06eaf750: 04dead28 06eaf78c 10001ec1 04dead28
0x06eaf760: 10e40030 000e1000 06eafb4c 026f9fb4
0x06eaf770: 04deac00 04dead28 6d92f61f 04dead28
0x06eaf780: 06eaf7d8 00000042 06eafb4c 026c65d0
0x06eaf790: 06eaf7d8 06eaf788 245d1968 06eafbf8
0x06eaf7a0: 245d19a0 ffffffff ffffffff 06eaf7e8

Instructions: (pc=0x026f86d0)
0x026f86b0: 8b b0 90 02 00 00 8b 41 14 8b 51 18 3b c2 0f 8d
0x026f86c0: 1b 00 00 00 8b d0 42 89 51 14 8b 71 08 8b 51 0c
0x026f86d0: 0f be 04 06 83 c4 38 5d 85 05 00 01 3d 00 c3 89
0x026f86e0: 74 24 28 89 4c 24 24 90 e9 41 00 00 00 e9 46 00


Register to memory mapping:

EAX=0x00001ca5 is an unknown value
EBX=0x00000036 is an unknown value
ECX=0x245d1968 is an oop
java.nio.DirectByteBuffer
- klass: 'java/nio/DirectByteBuffer'
EDX=0x00000000 is an unknown value
ESP=0x06eaf730 is pointing into the stack for thread: 0x04deac00
EBP=0x06eafb4c is pointing into the stack for thread: 0x04deac00
ESI=0x10e40030 is an unknown value
EDI=0x00001ca4 is an unknown value


Stack: [0x06e60000,0x06eb0000], sp=0x06eaf730, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J java.nio.DirectByteBuffer.get()B
V [jvm.dll+0xfac3b]
V [jvm.dll+0x18c3a1]
V [jvm.dll+0xfade1]
V [jvm.dll+0xfae3b]
V [jvm.dll+0xb5569]
V [jvm.dll+0x118f14]
V [jvm.dll+0x140ffc]
C [msvcr71.dll+0x9565] endthreadex+0xa0
C [kernel32.dll+0x1339a] BaseThreadInitThunk+0x12
C [ntdll.dll+0x39ef2] RtlInitializeExceptionChain+0x63
C [ntdll.dll+0x39ec5] RtlInitializeExceptionChain+0x36


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x04dfd400 JavaThread "Thread-15" [_thread_blocked, id=11144, stack(0x08a60000,0x08ab0000)]
0x04dfb800 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=13780, stack(0x071d0000,0x07220000)]
0x04dfb400 JavaThread "DestroyJavaVM" [_thread_blocked, id=10304, stack(0x00380000,0x003d0000)]
0x04dfac00 JavaThread "TimerQueue" daemon [_thread_blocked, id=9580, stack(0x07010000,0x07060000)]
0x04def800 JavaThread "Thread-9" [_thread_blocked, id=11720, stack(0x06ef0000,0x06f40000)]
=>0x04deac00 JavaThread "Camera Background" [_thread_in_Java, id=13392, stack(0x06e60000,0x06eb0000)]
0x04ddd400 JavaThread "Thread-4" [_thread_blocked, id=10464, stack(0x06b20000,0x06b70000)]
0x04dd5800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=8176, stack(0x072d0000,0x07320000)]
0x04d51800 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=12412, stack(0x050c0000,0x05110000)]
0x04c91000 JavaThread "AWT-Windows" daemon [_thread_in_native, id=1168, stack(0x05030000,0x05080000)]
0x025c6000 JavaThread "AWT-Shutdown" [_thread_blocked, id=12296, stack(0x04fa0000,0x04ff0000)]
0x025c0000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=14840, stack(0x04c00000,0x04c50000)]
0x0258a000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=10976, stack(0x049b0000,0x04a00000)]
0x02585400 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=11368, stack(0x04920000,0x04970000)]
0x02584000 JavaThread "Attach Listener" daemon [_thread_blocked, id=13576, stack(0x04890000,0x048e0000)]
0x02581000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11548, stack(0x04800000,0x04850000)]
0x02576c00 JavaThread "Finalizer" daemon [_thread_blocked, id=13976, stack(0x04770000,0x047c0000)]
0x02575800 JavaThread "Reference Handler" daemon [_thread_blocked, id=14888, stack(0x046e0000,0x04730000)]

Other Threads:
0x02537c00 VMThread [stack: 0x04650000,0x046a0000] [id=1900]
0x0259dc00 WatcherThread [stack: 0x04a40000,0x04a90000] [id=15080]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 78656K, used 906K [0x244f0000, 0x29a40000, 0x29a40000)
eden space 69952K, 1% used [0x244f0000, 0x245d2810, 0x28940000)
from space 8704K, 0% used [0x291c0000, 0x291c0000, 0x29a40000)
to space 8704K, 0% used [0x28940000, 0x28940000, 0x291c0000)
tenured generation total 174784K, used 140678K [0x29a40000, 0x344f0000, 0x344f0000)
the space 174784K, 80% used [0x29a40000, 0x323a1b10, 0x323a1c00, 0x344f0000)
compacting perm gen total 12288K, used 5004K [0x344f0000, 0x350f0000, 0x384f0000)
the space 12288K, 40% used [0x344f0000, 0x349d3080, 0x349d3200, 0x350f0000)
ro space 10240K, 51% used [0x384f0000, 0x38a1da30, 0x38a1dc00, 0x38ef0000)
rw space 12288K, 55% used [0x38ef0000, 0x39589b50, 0x39589c00, 0x39af0000)

Code Cache [0x025e0000, 0x027b8000, 0x045e0000)
total_blobs=1232 nmethods=988 adapters=179 free_code_cache=31637312 largest_free_block=0

[Jared Russell]

I'm going to try to use the latest OpenCV, JavaCV, and ffmpeg and see if it makes a difference.

I tried this, and it didn't make a difference on my machine.

[Mr. Lim]

Does this appear to be a memory leak to everyone else?

SmartDashboard progressively uses more and more memory as time goes on. It usually gets over 300MB before crashing.

Is anyone else noticing extremely high memory usage prior to the crashes?

[Jared Russell]

Does this appear to be a memory leak to everyone else?

SmartDashboard progressively uses more and more memory as time goes on. It usually gets over 300MB before crashing.

Is anyone else noticing extremely high memory usage prior to the crashes?

I am not entirely convinced it is a memory leak. You can start SmartDashboard with -Xms and -Xmx options to control the heap size, and I usually see the crash well before it hits the memory limit I specify.

[misko]

I'm not sure it's a memory leak. I got some additional debug information, but I'm no java expert. I think the problem is below. What happens if cam.getNewImage returns NULL? Because based on the traceback I got that's what appears to be happening. I don't see an explicit check to validate that we got an image (but I haven't looked at the getNewImage code), but if it does get a NULL, then it explains the access violation.
Any java coders out there that can help?

try {
image = cam.getNewImage(5.0);

if (image instanceof WPIColorImage) {
drawnImage = processImage((WPIColorImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);

} else if (image instanceof WPIGrayscaleImage) {
drawnImage = processImage((WPIGrayscaleImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);
}
} catch (final Exception e) {
e.printStackTrace();
cam.dispose();
cam = null;
drawnImage = null;
SwingUtilities.invokeLater(draw);
try {
Thread.sleep(2000);
} catch (InterruptedException ex) {
}
}

[Jared Russell]

There should not be a problem there. The Java "instanceof" operator will simply return false if the left-hand operand is null.

[JohnFogarty]

My CPU and processes go sky high until the file crashes, I even set my priority for java to the High level to see if it ever stopped using more & more memory.
I seriously think it's a memory leak.

[misko]

Hmm... I can watch the javavm in the windows task manager. The memory usage grows, then resets once garbage collection runs. So that's why I'm not sure it is a memory leak, but we may be running slightly different code.

I have been able to get it (smartdashboard) to run for several hours without the access violation. My problem is, I don't understand why. All I did was add a print statement, which when not running from command line displays nothing. But, it does seem to allow it to run for hours. So either it's slowed it down enough for a race condition to disappear, or it has changed the stack layout so that some memory corruption problem has moved or something I'm not thinking of, but it does seem to run better.

If someone else wants to try it to see if it fixes their access violation, drop me a private message and I can either tell you the one line I added to WPICameraExtension.java, or if you can't build it, I can send you a WPICameraExtension.jar file. No guarantees, but as I said, I now have had it running for several hours multiple times. It just bugs me that I can't explain it. If you can build it it may be easier as I've built against opencv installed in c:\opencv. That may or may not matter, but I should probably check before I give our a jar file that may have unresolved dependencies on a "standard" system.

[JohnFogarty]

I'd like to try your fix, I'm using my iPod to view the forums so I can't find a way to PM you, but I would like to try it.

[misko]

I copied the jar to another laptop we have with smartdashboard installed. It's been running 30 minutes now, never made it past 5 or 10 before. So I'm hopeful. Still wish I knew why this changes the behavior, but I'm no Java guru.
At least I know the jar doesn't have any odd dependencies.

[JohnFogarty]

your fix works for me, I didn't encounter a crash in over 2+ hours.

[agartner01]

Can I get a copy of this?

[loafdog]

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

[misko]

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

OK, but as I've said, I can't explain why this prevents the crash. It isn't the obvious thing validateDisposed, so it must be a timing thing.

In WPICameraExtension.java, around line 74 my code looks like this:

try {
image = cam.getNewImage(5.0);

if (image instanceof WPIColorImage) {
System.out.println("process image width:" + image.getWidth() + " height:" + image.getHeight());
drawnImage = processImage((WPIColorImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);

} else if (image instanceof WPIGrayscaleImage) {
drawnImage = processImage((WPIGrayscaleImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);
}
} catch (final Exception e) {

What I added was just before the call to processImage I print out the width and height of the image. 640x480 is what I get if I actually run from command line with a console, but if I just run First Dashboard I get no printouts.

Here is the 1 line print:
System.out.println("process image width:" + image.getWidth() + " height:" + image.getHeight());

Build it, make a new WPICameraExtension.jar, copy it to Program Files\SmartDashboard\extensions perhaps first saving the old jar file somewhere, and then just run it as usual.

The jar is also available at
http://athenian-arc.googlecode.com/files/WPICameraExtension.jar

[misko]

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

I posted, but included a link to the jar file, so I guess it is still waiting for approval. I'll just post the code changes later today if the original post doesn't show up.