First thing to look at is whether you're leaking heap. Enabling
-XX:+HeapDumpOnOutOfMemoryError and then using
jhat to analyze the result after you run out of memory may provide some information.
Note that I haven't run that procedure on Java on the roboRIO.