OpenCV Crashing?!

[Bradley Boxer]

We used GRIP to generate a vision pipeline (java), and then overlay the first two contours it sees onto a SmartDashboard camera feed. Last week, this was working great.

This week, we noticed sporadic and mysterious drops in comms with the RoboRio. When this happens, the driver station (and the computer) stops being able to ping the RoboRio until the robot is either rebooted or the Rio reset. The camera code has not changed from last week to now, but we have updated WPILib to 2017.2 and set up a PID loop on a CANTalon.

As to investigating the source of the error, we have USB'd into the Rio after a crash and looked at the FRC log, and there we found a java error with a message about further information at /tmp/hs_err_pid1429.log. The log at this location is as follows:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0xad323b9a, pid=1429, tid=2892047456
#
# JRE version: Java(TM) SE Embedded Runtime Environment (8.0_06-b23) (build 1.8.0_06-b23)
# Java VM: Java HotSpot(TM) Embedded Client VM (25.6-b23 mixed mode linux-arm )
# Problematic frame:
# C  [libopencv_core.so.3.1+0x14bb9a]  cv::_OutputArray::create(int, int const*, int, int, bool, int) const+0x8b1
#
# Core dump written. Default location: //core or core.1429 (max size 2048 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0xacd68000):  JavaThread "Thread-0" daemon [_thread_in_native, id=1468, stack(0xac5c3000,0xac613000)]

siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN), si_addr=0x00000056

Registers:
  r0  = 0x00000000
  r1  = 0x00de0000
  r2  = 0xac6117d0
  r3  = 0x00000056
  r4  = 0x0000001c
  r5  = 0xffffffff
  r6  = 0xac611988
  r7  = 0x00000002
  r8  = 0x00000001
  r9  = 0x0000003c
  r10 = 0xab345ef0
  fp  = 0x00000001
  r12 = 0xad381a2c
  sp  = 0xac611570
  lr  = 0xad323edf
  pc  = 0xad323b9a
  cpsr = 0x00070030

Top of Stack: (sp=0xac611570)
0xac611570:   00000000 00000000 ac6117d0 0000001c
0xac611580:   fffffffe 00000000 00000000 00000000
0xac611590:   ae071130 ab3a78d8 00000000 00000000
0xac6115a0:   00000000 00000002 00000000 00000167
0xac6115b0:   0000a7d0 ad10dc84 ad10dc84 ad10dc84
0xac6115c0:   00000000 00000000 ac6115a8 ac6115d0
0xac6115d0:   000001e0 00000001 42ff4000 00000002
0xac6115e0:   00000000 00000000 00000000 00000000 

Instructions: (pc=0xad323b9a)
0xad323b7a:   f8cd4649 f6d2c000 4648ed02 ec9cf6d2
0xad323b8a:   3024f8da 330cb173 8f5ff3bf 2f00e853
0xad323b9a:   e8431e51 28001000 2a01d1f8 8f5ff3bf
0xad323baa:   85e1f000 2004f8da f8ca2300 429a3024 

Register to memory mapping:

  r0  = 0x00000000
0x00000000 is an unknown value

  r1  = 0x00de0000
0x00de0000 is an unknown value

  r2  = 0xac6117d0
0xac6117d0 is pointing into the stack for thread: 0xacd68000

  r3  = 0x00000056
0x00000056 is an unknown value

  r4  = 0x0000001c
0x0000001c is an unknown value

  r5  = 0xffffffff
0xffffffff is an unknown value

  r6  = 0xac611988
0xac611988 is pointing into the stack for thread: 0xacd68000

  r7  = 0x00000002
0x00000002 is an unknown value

  r8  = 0x00000001
0x00000001 is an unknown value

  r9  = 0x0000003c
0x0000003c is an unknown value

  r10 = 0xab345ef0
0xab345ef0 is an unknown value

  fp  = 0x00000001
0x00000001 is an unknown value

  r12 = 0xad381a2c
0xad381a2c: <offset 0x1a9a2c> in /usr/local/frc/lib/libopencv_core.so.3.1 at 0xad1d8000

  sp  = 0xac611570
0xac611570 is pointing into the stack for thread: 0xacd68000

  lr  = 0xad323edf
0xad323edf: _ZNK2cv12_OutputArray6createEiPKiiibi+0xbf6 in /usr/local/frc/lib/libopencv_core.so.3.1 at 0xad1d8000

  pc  = 0xad323b9a
0xad323b9a: _ZNK2cv12_OutputArray6createEiPKiiibi+0x8b1 in /usr/local/frc/lib/libopencv_core.so.3.1 at 0xad1d8000



Stack: [0xac5c3000,0xac613000],  sp=0xac611570,  free space=313k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libopencv_core.so.3.1+0x14bb9a]  cv::_OutputArray::create(int, int const*, int, int, bool, int) const+0x8b1

I have never really had any experience with debugging this low level, and best I've been able to ascertain is that the library libopencv_core.so.3.1 is crashing. Any help would be greatly appreciated.

[balloman]

I don't know much about GRIP as we are using c++ OpenCV, but it looks like when you drop the frames, OpenCV gets images that it can't use and it crashes the program. Is there anything in particular you are doing with the bot before it crahes?

[Bradley Boxer]

90% if the times it crashes, the robot is just standing still. Sometimes it is pointed at retroreflective tape (so it tries to draw contours) but sometimes it is not.

Our camera has been dropping out once in awhile (for about 500ms here and there), but I didn't think it would matter, since our code continues the thread before the pipeline if cvSink.grabFrame(mat) is equal to zero.

Additionally the Rio doesn't crash every time the camera drops, so this didn't make sense to me as the cause of the issue. I could be wrong, though.

[SamCarlberg]

Can you share your project? A zip of the eclipse project or a link to a commit on github would be fine

[Bradley Boxer]

Our GitHub link to our last camera code update is https://github.com/Team303/Java-Robo...986e305d8e4b52. The class running camera and OpenCV code is Camera, and our generated pipeline is BoilerPipeline.

[billbo911]

I have seen OpenCV projects crash many times before, but I have never run them on the RoboRio. When they crash, there is usually a solid reason. Sometimes though, it appears random, but when you get to the root of it, it also becomes solid as well.

Out of curiosity, what USB camera make and model are you using?

[Bradley Boxer]

We are using an IP Camera (Axis M1011).