SmartDashboard Crashing

[Brian Selle]

Love the SmartDashboard but we are having the same issue. We don't do any image processing using the JavaCV libraries, just display the camera feed. The JVM dump file(hs_err_pidxxxxx.log) in program files/SmartDashboard looks like:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x026f86d0, pid=13336, tid=13392
#
# JRE version: 6.0_30-b12
# Java VM: Java HotSpot(TM) Client VM (20.5-b03 mixed mode, sharing windows-x86 )
# Problematic frame:
# J java.nio.DirectByteBuffer.get()B
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x04deac00): JavaThread "Camera Background" [_thread_in_Java, id=13392, stack(0x06e60000,0x06eb0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x10e41cd5

Registers:
EAX=0x00001ca5, EBX=0x00000036, ECX=0x245d1968, EDX=0x00000000
ESP=0x06eaf730, EBP=0x06eafb4c, ESI=0x10e40030, EDI=0x00001ca4
EIP=0x026f86d0, EFLAGS=0x00010207

Top of Stack: (sp=0x06eaf730)
0x06eaf730: 06eaf754 6d88ea06 04dead28 04d52134
0x06eaf740: 04d51fc8 10e40030 00000000 000e1000
0x06eaf750: 04dead28 06eaf78c 10001ec1 04dead28
0x06eaf760: 10e40030 000e1000 06eafb4c 026f9fb4
0x06eaf770: 04deac00 04dead28 6d92f61f 04dead28
0x06eaf780: 06eaf7d8 00000042 06eafb4c 026c65d0
0x06eaf790: 06eaf7d8 06eaf788 245d1968 06eafbf8
0x06eaf7a0: 245d19a0 ffffffff ffffffff 06eaf7e8

Instructions: (pc=0x026f86d0)
0x026f86b0: 8b b0 90 02 00 00 8b 41 14 8b 51 18 3b c2 0f 8d
0x026f86c0: 1b 00 00 00 8b d0 42 89 51 14 8b 71 08 8b 51 0c
0x026f86d0: 0f be 04 06 83 c4 38 5d 85 05 00 01 3d 00 c3 89
0x026f86e0: 74 24 28 89 4c 24 24 90 e9 41 00 00 00 e9 46 00


Register to memory mapping:

EAX=0x00001ca5 is an unknown value
EBX=0x00000036 is an unknown value
ECX=0x245d1968 is an oop
java.nio.DirectByteBuffer
- klass: 'java/nio/DirectByteBuffer'
EDX=0x00000000 is an unknown value
ESP=0x06eaf730 is pointing into the stack for thread: 0x04deac00
EBP=0x06eafb4c is pointing into the stack for thread: 0x04deac00
ESI=0x10e40030 is an unknown value
EDI=0x00001ca4 is an unknown value


Stack: [0x06e60000,0x06eb0000], sp=0x06eaf730, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J java.nio.DirectByteBuffer.get()B
V [jvm.dll+0xfac3b]
V [jvm.dll+0x18c3a1]
V [jvm.dll+0xfade1]
V [jvm.dll+0xfae3b]
V [jvm.dll+0xb5569]
V [jvm.dll+0x118f14]
V [jvm.dll+0x140ffc]
C [msvcr71.dll+0x9565] endthreadex+0xa0
C [kernel32.dll+0x1339a] BaseThreadInitThunk+0x12
C [ntdll.dll+0x39ef2] RtlInitializeExceptionChain+0x63
C [ntdll.dll+0x39ec5] RtlInitializeExceptionChain+0x36


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x04dfd400 JavaThread "Thread-15" [_thread_blocked, id=11144, stack(0x08a60000,0x08ab0000)]
0x04dfb800 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=13780, stack(0x071d0000,0x07220000)]
0x04dfb400 JavaThread "DestroyJavaVM" [_thread_blocked, id=10304, stack(0x00380000,0x003d0000)]
0x04dfac00 JavaThread "TimerQueue" daemon [_thread_blocked, id=9580, stack(0x07010000,0x07060000)]
0x04def800 JavaThread "Thread-9" [_thread_blocked, id=11720, stack(0x06ef0000,0x06f40000)]
=>0x04deac00 JavaThread "Camera Background" [_thread_in_Java, id=13392, stack(0x06e60000,0x06eb0000)]
0x04ddd400 JavaThread "Thread-4" [_thread_blocked, id=10464, stack(0x06b20000,0x06b70000)]
0x04dd5800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=8176, stack(0x072d0000,0x07320000)]
0x04d51800 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=12412, stack(0x050c0000,0x05110000)]
0x04c91000 JavaThread "AWT-Windows" daemon [_thread_in_native, id=1168, stack(0x05030000,0x05080000)]
0x025c6000 JavaThread "AWT-Shutdown" [_thread_blocked, id=12296, stack(0x04fa0000,0x04ff0000)]
0x025c0000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=14840, stack(0x04c00000,0x04c50000)]
0x0258a000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=10976, stack(0x049b0000,0x04a00000)]
0x02585400 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=11368, stack(0x04920000,0x04970000)]
0x02584000 JavaThread "Attach Listener" daemon [_thread_blocked, id=13576, stack(0x04890000,0x048e0000)]
0x02581000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11548, stack(0x04800000,0x04850000)]
0x02576c00 JavaThread "Finalizer" daemon [_thread_blocked, id=13976, stack(0x04770000,0x047c0000)]
0x02575800 JavaThread "Reference Handler" daemon [_thread_blocked, id=14888, stack(0x046e0000,0x04730000)]

Other Threads:
0x02537c00 VMThread [stack: 0x04650000,0x046a0000] [id=1900]
0x0259dc00 WatcherThread [stack: 0x04a40000,0x04a90000] [id=15080]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 78656K, used 906K [0x244f0000, 0x29a40000, 0x29a40000)
eden space 69952K, 1% used [0x244f0000, 0x245d2810, 0x28940000)
from space 8704K, 0% used [0x291c0000, 0x291c0000, 0x29a40000)
to space 8704K, 0% used [0x28940000, 0x28940000, 0x291c0000)
tenured generation total 174784K, used 140678K [0x29a40000, 0x344f0000, 0x344f0000)
the space 174784K, 80% used [0x29a40000, 0x323a1b10, 0x323a1c00, 0x344f0000)
compacting perm gen total 12288K, used 5004K [0x344f0000, 0x350f0000, 0x384f0000)
the space 12288K, 40% used [0x344f0000, 0x349d3080, 0x349d3200, 0x350f0000)
ro space 10240K, 51% used [0x384f0000, 0x38a1da30, 0x38a1dc00, 0x38ef0000)
rw space 12288K, 55% used [0x38ef0000, 0x39589b50, 0x39589c00, 0x39af0000)

Code Cache [0x025e0000, 0x027b8000, 0x045e0000)
total_blobs=1232 nmethods=988 adapters=179 free_code_cache=31637312 largest_free_block=0

[Jared Russell]

Originally Posted by misko

I'm going to try to use the latest OpenCV, JavaCV, and ffmpeg and see if it makes a difference.

I tried this, and it didn't make a difference on my machine.

[Mr. Lim]

Does this appear to be a memory leak to everyone else?

SmartDashboard progressively uses more and more memory as time goes on. It usually gets over 300MB before crashing.

Is anyone else noticing extremely high memory usage prior to the crashes?

[Jared Russell]

Originally Posted by Mr. Lim

Does this appear to be a memory leak to everyone else?

SmartDashboard progressively uses more and more memory as time goes on. It usually gets over 300MB before crashing.

Is anyone else noticing extremely high memory usage prior to the crashes?

I am not entirely convinced it is a memory leak. You can start SmartDashboard with -Xms and -Xmx options to control the heap size, and I usually see the crash well before it hits the memory limit I specify.

[misko]

I'm not sure it's a memory leak. I got some additional debug information, but I'm no java expert. I think the problem is below. What happens if cam.getNewImage returns NULL? Because based on the traceback I got that's what appears to be happening. I don't see an explicit check to validate that we got an image (but I haven't looked at the getNewImage code), but if it does get a NULL, then it explains the access violation.
Any java coders out there that can help?

try {
image = cam.getNewImage(5.0);

if (image instanceof WPIColorImage) {
drawnImage = processImage((WPIColorImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);

} else if (image instanceof WPIGrayscaleImage) {
drawnImage = processImage((WPIGrayscaleImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);
}
} catch (final Exception e) {
e.printStackTrace();
cam.dispose();
cam = null;
drawnImage = null;
SwingUtilities.invokeLater(draw);
try {
Thread.sleep(2000);
} catch (InterruptedException ex) {
}
}

[Jared Russell]

There should not be a problem there. The Java "instanceof" operator will simply return false if the left-hand operand is null.

[JohnFogarty]

My CPU and processes go sky high until the file crashes, I even set my priority for java to the High level to see if it ever stopped using more & more memory.
I seriously think it's a memory leak.

[misko]

Hmm... I can watch the javavm in the windows task manager. The memory usage grows, then resets once garbage collection runs. So that's why I'm not sure it is a memory leak, but we may be running slightly different code.

I have been able to get it (smartdashboard) to run for several hours without the access violation. My problem is, I don't understand why. All I did was add a print statement, which when not running from command line displays nothing. But, it does seem to allow it to run for hours. So either it's slowed it down enough for a race condition to disappear, or it has changed the stack layout so that some memory corruption problem has moved or something I'm not thinking of, but it does seem to run better.

If someone else wants to try it to see if it fixes their access violation, drop me a private message and I can either tell you the one line I added to WPICameraExtension.java, or if you can't build it, I can send you a WPICameraExtension.jar file. No guarantees, but as I said, I now have had it running for several hours multiple times. It just bugs me that I can't explain it. If you can build it it may be easier as I've built against opencv installed in c:\opencv. That may or may not matter, but I should probably check before I give our a jar file that may have unresolved dependencies on a "standard" system.

[JohnFogarty]

I'd like to try your fix, I'm using my iPod to view the forums so I can't find a way to PM you, but I would like to try it.

[misko]

I copied the jar to another laptop we have with smartdashboard installed. It's been running 30 minutes now, never made it past 5 or 10 before. So I'm hopeful. Still wish I knew why this changes the behavior, but I'm no Java guru.
At least I know the jar doesn't have any odd dependencies.

[JohnFogarty]

your fix works for me, I didn't encounter a crash in over 2+ hours.

[agartner01]

Can I get a copy of this?

[loafdog]

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

[misko]

Originally Posted by loafdog

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

OK, but as I've said, I can't explain why this prevents the crash. It isn't the obvious thing validateDisposed, so it must be a timing thing.

In WPICameraExtension.java, around line 74 my code looks like this:

try {
image = cam.getNewImage(5.0);

if (image instanceof WPIColorImage) {
System.out.println("process image width:" + image.getWidth() + " height:" + image.getHeight());
drawnImage = processImage((WPIColorImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);

} else if (image instanceof WPIGrayscaleImage) {
drawnImage = processImage((WPIGrayscaleImage) image).getBufferedImage();
SwingUtilities.invokeLater(draw);
}
} catch (final Exception e) {

What I added was just before the call to processImage I print out the width and height of the image. 640x480 is what I get if I actually run from command line with a console, but if I just run First Dashboard I get no printouts.

Here is the 1 line print:
System.out.println("process image width:" + image.getWidth() + " height:" + image.getHeight());

Build it, make a new WPICameraExtension.jar, copy it to Program Files\SmartDashboard\extensions perhaps first saving the old jar file somewhere, and then just run it as usual.

The jar is also available at
http://athenian-arc.googlecode.com/f...aExtension.jar

[misko]

Originally Posted by loafdog

Please post your changes here so everyone can see/try it out.
Our smartdashboard is crashing as well after we added the camera widget.

I posted, but included a link to the jar file, so I guess it is still waiting for approval. I'll just post the code changes later today if the original post doesn't show up.