Memory Leaks

truher · August 6, 2023, 5:13am

hey everybody,

team 100 has been struggling with memory issues recently, and so i’ve been looking into it.

team 100 is a java team. there are several different symptoms; sometimes the JRE can’t mmap a chunk of memory; sometimes the error appears somewhere else, like it fails during GC, or it fails during instantiation. i looked at the VM with JConsole, and didn’t find anything interesting. one interesting thing about the failure is that it seems to survive deployments: once the roborio is “sick” the only thing that can cure it is a “reset” or power cycle.

One thing that seems to relate to the problem is the use of non-existent CAN devices, e.g. a motor controller that’s not actually connected. this produces various error messages (e.g. “CAN frame not received” from CTRE and “CAN output buffer full” from REV). are these errors perhaps leaking something in C+±land that JConsole can’t see? are there tools I could use to see the memory in there? does this sound like a familiar problem to anyone?

Peter_Johnson · August 6, 2023, 5:40am

Since it clears with a reboot, there’s a few different potential causes:

Lots of text output to stdout/stderr (eg CAN not found errors) can consume available memory until a reboot. The console output is saved to a file in /var/log, and this is a memory backed filesystem on the Rio. This is the most likely culprit if you know you have lots of console output.
I have seen memory leaks at times in some of the built-in background processes, including the system web server and NetComm. But this isn’t as common as the first one.

truher · August 6, 2023, 2:44pm

thanks! I checked the usage of the volatile filesystem and that’s definitely not the issue. there’s not zero output by it’s not that much, maybe 1kb by the time it starts crashing.

also none of the other processes are leaking, at least according to top.

another interesting clue, the problem exists even if the Robot constructor sleeps forever after instantiating the container, so its not something our code is touching directly.

nuttle · August 7, 2023, 1:17am

Given that you ran top, it should be pretty easy to do something like `cat /proc/meminfo’ and pasting this here. It sounds as though you have this pinned down to a particular process, but this might contain a clue of some type. Even posting what top is showing might be helpful…

truher · August 7, 2023, 2:25am

sure, that’s a good idea. so here’s the line in top:

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 7611  7600 lvuser   S     260m 106%  57% /usr/local/frc/JRE/bin/java ...

here’s its smap rollup

00010000-ffff1000 ---p 00000000 00:00 0          [rollup]
Rss:               69884 kB
Pss:               64352 kB
Shared_Clean:       7024 kB
Shared_Dirty:          0 kB
Private_Clean:     16324 kB
Private_Dirty:     46536 kB
Referenced:        69884 kB
Anonymous:         45984 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB

and the machine meminfo:

MemTotal:         250152 kB
MemFree:            3720 kB
MemAvailable:     108508 kB
Buffers:               0 kB
Cached:           126560 kB
SwapCached:            0 kB
Active:           139464 kB
Inactive:          31172 kB
Active(anon):      67648 kB
Inactive(anon):      640 kB
Active(file):      71816 kB
Inactive(file):    30532 kB
Unevictable:       36936 kB
Mlocked:           36936 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         250152 kB
LowFree:            3720 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             4 kB
AnonPages:         81036 kB
Mapped:            49268 kB
Shmem:              1236 kB
Slab:              19760 kB
SReclaimable:      10924 kB
SUnreclaim:         8836 kB
KernelStack:        1720 kB
PageTables:         1408 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      215128 kB
Committed_AS:     172868 kB
VmallocTotal:     770048 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB

and here’s the status. note the weirdly high VmPeak.

Name:   java
Umask:  0022
State:  S (sleeping)
Tgid:   7611
Ngid:   0
Pid:    7611
PPid:   7600
TracerPid:      0
Uid:    500     500     500     500
Gid:    500     500     500     500
FDSize: 256
Groups: 5 44 46 397 498 500
VmPeak:  2971248 kB
VmSize:   266508 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     72136 kB
VmRSS:     72136 kB
RssAnon:           48652 kB
RssFile:           22940 kB
RssShmem:            544 kB
VmData:    96284 kB
VmStk:       132 kB
VmExe:         4 kB
VmLib:     38228 kB
VmPTE:       136 kB
VmPMD:         0 kB
VmSwap:        0 kB
Threads:        51
SigQ:   0/1946
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000003000
SigCgt: 2000000181004ccf
CapInh: 0000000000000000
CapPrm: 0000000000800000
CapEff: 0000000000800000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Speculation_Store_Bypass:       unknown
Cpus_allowed:   3
Cpus_allowed_list:      0-1
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        1
nonvoluntary_ctxt_switches:     5

So the main puzzling thing is that, in the “sick” mode, there is plenty of physical RAM available, yet even small mallocs fail. It’s like the actual footprint of the process is somehow larger than the RSS, or there’s some other limit being imposed. Often the memory error occurs during JIT compilation so i tried turning that off, and also tried compiling everything in batch, but it wasn’t any better.

I produced a heap dump (using jmx) and looked at it with VisualVM. It was both unremarkable and tiny, under 4MB.

Looking at the proc maps (below, edited a bit to stay under the 32k message limit) there sure is a lot of opencv code that i don’t need. is there an easy way to remove it? it looks like maybe just kind of inlining and modifying the gradle functions like wpilibJniRelease would work?

Anyway if you’ve read this far, thank you.

00010000-00011000 r-xp 00000000 00:0e 11450      /usr/local/frc/JRE/bin/java
00020000-00021000 r--p 00000000 00:0e 11450      /usr/local/frc/JRE/bin/java
00021000-00022000 rw-p 00001000 00:0e 11450      /usr/local/frc/JRE/bin/java
01767000-017ec000 rw-p 00000000 00:00 0          [heap]
a6d00000-a6d39000 rw-p 00000000 00:00 0
...
a703c000-a703f000 ---p 00000000 00:00 0
a703f000-a708c000 rw-p 00000000 00:00 0
a708c000-a71ee000 r-xp 00000000 00:0e 11137      /usr/local/frc/third-party/lib/libwpimath.so
a71ee000-a71fd000 ---p 00162000 00:0e 11137      /usr/local/frc/third-party/lib/libwpimath.so
a71fd000-a71fe000 r--p 00161000 00:0e 11137      /usr/local/frc/third-party/lib/libwpimath.so
a71fe000-a7200000 rw-p 00162000 00:0e 11137      /usr/local/frc/third-party/lib/libwpimath.so
a7200000-a7267000 rw-p 00000000 00:00 0
a7267000-a7300000 ---p 00000000 00:00 0
...
a7900000-a79b0000 rw-p 00000000 00:00 0
a79b0000-a7c00000 ---p 00000000 00:00 0
a7c0a000-a7cee000 r-xp 00000000 00:0e 11139      /usr/local/frc/third-party/lib/libopencv_java460.so
a7cee000-a7cfd000 ---p 000e4000 00:0e 11139      /usr/local/frc/third-party/lib/libopencv_java460.so
a7cfd000-a7cff000 r--p 000e3000 00:0e 11139      /usr/local/frc/third-party/lib/libopencv_java460.so
a7cff000-a7d00000 rw-p 000e5000 00:0e 11139      /usr/local/frc/third-party/lib/libopencv_java460.so
a7d00000-a7dfd000 rw-p 00000000 00:00 0
a7dfd000-a7e00000 ---p 00000000 00:00 0
a7e20000-a7e90000 r-xp 00000000 00:0e 11121      /usr/local/frc/third-party/lib/libwpimathjni.so
a7e90000-a7e9f000 ---p 00070000 00:0e 11121      /usr/local/frc/third-party/lib/libwpimathjni.so
a7e9f000-a7ea0000 r--p 0006f000 00:0e 11121      /usr/local/frc/third-party/lib/libwpimathjni.so
a7ea0000-a7ea1000 rw-p 00070000 00:0e 11121      /usr/local/frc/third-party/lib/libwpimathjni.so
a7ea1000-a7ea4000 ---p 00000000 00:00 0
...
a9fc2000-aa001000 rw-p 00000000 00:00 0
aa001000-aa21c000 r-xp 00000000 00:0e 11145      /usr/local/frc/third-party/lib/libopencv_core.so.4.6
aa21c000-aa22c000 ---p 0021b000 00:0e 11145      /usr/local/frc/third-party/lib/libopencv_core.so.4.6
aa22c000-aa232000 r--p 0021b000 00:0e 11145      /usr/local/frc/third-party/lib/libopencv_core.so.4.6
aa232000-aa233000 rw-p 00221000 00:0e 11145      /usr/local/frc/third-party/lib/libopencv_core.so.4.6
aa233000-aa236000 rw-p 00000000 00:00 0
aa236000-aa288000 r-xp 00000000 00:0e 11122      /usr/local/frc/third-party/lib/libopencv_objdetect.so.4.6
aa288000-aa298000 ---p 00052000 00:0e 11122      /usr/local/frc/third-party/lib/libopencv_objdetect.so.4.6
aa298000-aa29a000 r--p 00052000 00:0e 11122      /usr/local/frc/third-party/lib/libopencv_objdetect.so.4.6
aa29a000-aa29b000 rw-p 00054000 00:0e 11122      /usr/local/frc/third-party/lib/libopencv_objdetect.so.4.6
aa29b000-aa35b000 r-xp 00000000 00:0e 11128      /usr/local/frc/third-party/lib/libopencv_imgcodecs.so.4.6
aa35b000-aa36a000 ---p 000c0000 00:0e 11128      /usr/local/frc/third-party/lib/libopencv_imgcodecs.so.4.6
aa36a000-aa36d000 r--p 000bf000 00:0e 11128      /usr/local/frc/third-party/lib/libopencv_imgcodecs.so.4.6
aa36d000-aa36e000 rw-p 000c2000 00:0e 11128      /usr/local/frc/third-party/lib/libopencv_imgcodecs.so.4.6
aa36e000-aa3ae000 r-xp 00000000 00:0e 11146      /usr/local/frc/third-party/lib/libopencv_videoio.so.4.6
aa3ae000-aa3be000 ---p 00040000 00:0e 11146      /usr/local/frc/third-party/lib/libopencv_videoio.so.4.6
aa3be000-aa3bf000 r--p 00040000 00:0e 11146      /usr/local/frc/third-party/lib/libopencv_videoio.so.4.6
aa3bf000-aa3c0000 rw-p 00041000 00:0e 11146      /usr/local/frc/third-party/lib/libopencv_videoio.so.4.6
aa3c0000-aa3c2000 rw-p 00000000 00:00 0
aa3c2000-aa418000 r-xp 00000000 00:0e 11134      /usr/local/frc/third-party/lib/libopencv_ml.so.4.6
aa418000-aa427000 ---p 00056000 00:0e 11134      /usr/local/frc/third-party/lib/libopencv_ml.so.4.6
aa427000-aa429000 r--p 00055000 00:0e 11134      /usr/local/frc/third-party/lib/libopencv_ml.so.4.6
aa429000-aa42a000 rw-p 00057000 00:0e 11134      /usr/local/frc/third-party/lib/libopencv_ml.so.4.6
aa42a000-aa69d000 r-xp 00000000 00:0e 11136      /usr/local/frc/third-party/lib/libopencv_imgproc.so.4.6
aa69d000-aa6ad000 ---p 00273000 00:0e 11136      /usr/local/frc/third-party/lib/libopencv_imgproc.so.4.6
aa6ad000-aa6b5000 r--p 00273000 00:0e 11136      /usr/local/frc/third-party/lib/libopencv_imgproc.so.4.6
aa6b5000-aa6b9000 rw-p 0027b000 00:0e 11136      /usr/local/frc/third-party/lib/libopencv_imgproc.so.4.6
aa6b9000-aa755000 rw-p 00000000 00:00 0
aa755000-aa872000 r-xp 00000000 00:0e 11119      /usr/local/frc/third-party/lib/libopencv_calib3d.so.4.6
aa872000-aa881000 ---p 0011d000 00:0e 11119      /usr/local/frc/third-party/lib/libopencv_calib3d.so.4.6
aa881000-aa886000 r--p 0011c000 00:0e 11119      /usr/local/frc/third-party/lib/libopencv_calib3d.so.4.6
aa886000-aa887000 rw-p 00121000 00:0e 11119      /usr/local/frc/third-party/lib/libopencv_calib3d.so.4.6
aa887000-aa8da000 r-xp 00000000 00:0e 11144      /usr/local/frc/third-party/lib/libopencv_stitching.so.4.6
aa8da000-aa8ea000 ---p 00053000 00:0e 11144      /usr/local/frc/third-party/lib/libopencv_stitching.so.4.6
aa8ea000-aa8ec000 r--p 00053000 00:0e 11144      /usr/local/frc/third-party/lib/libopencv_stitching.so.4.6
aa8ec000-aa8ed000 rw-p 00055000 00:0e 11144      /usr/local/frc/third-party/lib/libopencv_stitching.so.4.6
aa8ed000-aacc9000 rw-p 00000000 00:00 0
aaccb000-aad07000 r-xp 00000000 00:0e 11148      /usr/local/frc/third-party/lib/libopencv_video.so.4.6
aad07000-aad16000 ---p 0003c000 00:0e 11148      /usr/local/frc/third-party/lib/libopencv_video.so.4.6
aad16000-aad18000 r--p 0003b000 00:0e 11148      /usr/local/frc/third-party/lib/libopencv_video.so.4.6
aad18000-aad19000 rw-p 0003d000 00:0e 11148      /usr/local/frc/third-party/lib/libopencv_video.so.4.6
aad19000-aad96000 r-xp 00000000 00:0e 11120      /usr/local/frc/third-party/lib/libopencv_photo.so.4.6
aad96000-aada5000 ---p 0007d000 00:0e 11120      /usr/local/frc/third-party/lib/libopencv_photo.so.4.6
aada5000-aada7000 r--p 0007c000 00:0e 11120      /usr/local/frc/third-party/lib/libopencv_photo.so.4.6
aada7000-aada8000 rw-p 0007e000 00:0e 11120      /usr/local/frc/third-party/lib/libopencv_photo.so.4.6
aada8000-aadee000 r-xp 00000000 00:0e 11130      /usr/local/frc/third-party/lib/libopencv_flann.so.4.6
aadee000-aadfd000 ---p 00046000 00:0e 11130      /usr/local/frc/third-party/lib/libopencv_flann.so.4.6
aadfd000-aadff000 r--p 00045000 00:0e 11130      /usr/local/frc/third-party/lib/libopencv_flann.so.4.6
aadff000-aae00000 rw-p 00047000 00:0e 11130      /usr/local/frc/third-party/lib/libopencv_flann.so.4.6
aae00000-ab200000 rw-s 00000000 00:06 10385      /dev/ni/atomiczynqk:0\atomiczynqk_atomicrioddkzynq\0
ab200000-ab2fc000 rw-p 00000000 00:00 0
ab2fc000-ab300000 ---p 00000000 00:00 0
ab300000-ab4f1000 rw-p 00000000 00:00 0
ab4f1000-ab500000 ---p 00000000 00:00 0
ab51b000-ab53b000 r-xp 00000000 00:0e 11142      /usr/local/frc/third-party/lib/libopencv_highgui.so.4.6
ab53b000-ab54b000 ---p 00020000 00:0e 11142      /usr/local/frc/third-party/lib/libopencv_highgui.so.4.6
ab54b000-ab54c000 r--p 00020000 00:0e 11142      /usr/local/frc/third-party/lib/libopencv_highgui.so.4.6
ab54c000-ab54d000 rw-p 00021000 00:0e 11142      /usr/local/frc/third-party/lib/libopencv_highgui.so.4.6
ab54d000-ab5ab000 r-xp 00000000 00:0e 11140      /usr/local/frc/third-party/lib/libopencv_features2d.so.4.6
ab5ab000-ab5bb000 ---p 0005e000 00:0e 11140      /usr/local/frc/third-party/lib/libopencv_features2d.so.4.6
ab5bb000-ab5be000 r--p 0005e000 00:0e 11140      /usr/local/frc/third-party/lib/libopencv_features2d.so.4.6
ab5be000-ab5c0000 rw-p 00061000 00:0e 11140      /usr/local/frc/third-party/lib/libopencv_features2d.so.4.6
ab5c0000-ab5e0000 rw-s 01000000 00:06 10385      /dev/ni/atomiczynqk:0\atomiczynqk_atomicrioddkzynq\0
ab5e0000-ab5e1000 ---p 00000000 00:00 0
...
ab6a1000-ab6e0000 rw-p 00000000 00:00 0
ab6e0000-ab78f000 r-xp 00000000 00:0e 4267       /usr/lib/arm-linux-gnueabi/libatomiczynqu.so.22.5.0
ab78f000-ab796000 ---p 000af000 00:0e 4267       /usr/lib/arm-linux-gnueabi/libatomiczynqu.so.22.5.0
ab796000-ab79f000 rw-p 000ae000 00:0e 4267       /usr/lib/arm-linux-gnueabi/libatomiczynqu.so.22.5.0
ab7ba000-ab7e4000 r-xp 00000000 00:0e 11127      /usr/local/frc/third-party/lib/libopencv_aruco.so.4.6
ab7e4000-ab7f3000 ---p 0002a000 00:0e 11127      /usr/local/frc/third-party/lib/libopencv_aruco.so.4.6
ab7f3000-ab7f4000 r--p 00029000 00:0e 11127      /usr/local/frc/third-party/lib/libopencv_aruco.so.4.6
ab7f4000-ab818000 rw-p 0002a000 00:0e 11127      /usr/local/frc/third-party/lib/libopencv_aruco.so.4.6
ab818000-ab819000 rw-p 00000000 00:00 0
ab819000-aba6f000 r-xp 00000000 00:0e 11129      /usr/local/frc/third-party/lib/libopencv_gapi.so.4.6
aba6f000-aba7e000 ---p 00256000 00:0e 11129      /usr/local/frc/third-party/lib/libopencv_gapi.so.4.6
aba7e000-aba84000 r--p 00255000 00:0e 11129      /usr/local/frc/third-party/lib/libopencv_gapi.so.4.6
aba84000-aba86000 rw-p 0025b000 00:0e 11129      /usr/local/frc/third-party/lib/libopencv_gapi.so.4.6
aba86000-abb1d000 r-xp 00000000 00:0e 11147      /usr/local/frc/third-party/lib/libcscore.so
abb1d000-abb2c000 ---p 00097000 00:0e 11147      /usr/local/frc/third-party/lib/libcscore.so
abb2c000-abb2d000 r--p 00096000 00:0e 11147      /usr/local/frc/third-party/lib/libcscore.so
abb2d000-abb2e000 rw-p 00097000 00:0e 11147      /usr/local/frc/third-party/lib/libcscore.so
abb2e000-abb43000 r-xp 00000000 00:0e 11138      /usr/local/frc/third-party/lib/libcscorejni.so
abb43000-abb52000 ---p 00015000 00:0e 11138      /usr/local/frc/third-party/lib/libcscorejni.so
abb52000-abb53000 r--p 00014000 00:0e 11138      /usr/local/frc/third-party/lib/libcscorejni.so
abb53000-abb54000 rw-p 00015000 00:0e 11138      /usr/local/frc/third-party/lib/libcscorejni.so
abb54000-abb5c000 r-xp 00000000 00:0e 11125      /usr/local/frc/third-party/lib/libwpiutiljni.so
abb5c000-abb6b000 ---p 00008000 00:0e 11125      /usr/local/frc/third-party/lib/libwpiutiljni.so
abb6b000-abb6c000 r--p 00007000 00:0e 11125      /usr/local/frc/third-party/lib/libwpiutiljni.so
abb6c000-abb6d000 rw-p 00008000 00:0e 11125      /usr/local/frc/third-party/lib/libwpiutiljni.so
abb6d000-abc66000 r-xp 00000000 00:0e 11133      /usr/local/frc/third-party/lib/libwpinet.so
abc66000-abc75000 ---p 000f9000 00:0e 11133      /usr/local/frc/third-party/lib/libwpinet.so
abc75000-abc78000 r--p 000f8000 00:0e 11133      /usr/local/frc/third-party/lib/libwpinet.so
abc78000-abc7a000 rw-p 000fb000 00:0e 11133      /usr/local/frc/third-party/lib/libwpinet.so
abc7a000-abc7b000 rw-p 00000000 00:00 0
abc7b000-abd75000 r-xp 00000000 00:0e 11124      /usr/local/frc/third-party/lib/libntcore.so
abd75000-abd85000 ---p 000fa000 00:0e 11124      /usr/local/frc/third-party/lib/libntcore.so
abd85000-abd87000 r--p 000fa000 00:0e 11124      /usr/local/frc/third-party/lib/libntcore.so
abd87000-abd89000 rw-p 000fc000 00:0e 11124      /usr/local/frc/third-party/lib/libntcore.so
abd89000-abda4000 r-xp 00000000 00:0e 11143      /usr/local/frc/third-party/lib/libntcorejni.so
abda4000-abdb4000 ---p 0001b000 00:0e 11143      /usr/local/frc/third-party/lib/libntcorejni.so
abdb4000-abdb5000 r--p 0001b000 00:0e 11143      /usr/local/frc/third-party/lib/libntcorejni.so
abdb5000-abdb6000 rw-p 0001c000 00:0e 11143      /usr/local/frc/third-party/lib/libntcorejni.so
abdb6000-abdb7000 ---p 00000000 00:00 0
...
abeb7000-abef6000 rw-p 00000000 00:00 0
abef6000-abf76000 rw-s 00000000 00:05 131072     /SYSV020e1074 (deleted)
abf76000-ac02f000 r-xp 00000000 00:0e 4260       /usr/lib/arm-linux-gnueabi/libniriodevenum.so.22.5.0
ac02f000-ac037000 ---p 000b9000 00:0e 4260       /usr/lib/arm-linux-gnueabi/libniriodevenum.so.22.5.0
ac037000-ac03a000 rw-p 000b9000 00:0e 4260       /usr/lib/arm-linux-gnueabi/libniriodevenum.so.22.5.0
ac03a000-ac106000 r-xp 00000000 00:0e 4238       /usr/lib/arm-linux-gnueabi/libNiRioSrv.so.22.5.0
ac106000-ac10d000 ---p 000cc000 00:0e 4238       /usr/lib/arm-linux-gnueabi/libNiRioSrv.so.22.5.0
ac10d000-ac113000 rw-p 000cb000 00:0e 4238       /usr/lib/arm-linux-gnueabi/libNiRioSrv.so.22.5.0
ac113000-ac155000 r-xp 00000000 00:0e 4248       /usr/lib/arm-linux-gnueabi/libNiFpga.so.22.5.0
ac155000-ac15d000 ---p 00042000 00:0e 4248       /usr/lib/arm-linux-gnueabi/libNiFpga.so.22.5.0
ac15d000-ac15e000 rw-p 00042000 00:0e 4248       /usr/lib/arm-linux-gnueabi/libNiFpga.so.22.5.0
ac15e000-ac187000 r-xp 00000000 00:0e 6502       /usr/local/natinst/lib/libni_emb.so.13.0.0
ac187000-ac18f000 ---p 00029000 00:0e 6502       /usr/local/natinst/lib/libni_emb.so.13.0.0
ac18f000-ac190000 rw-p 00029000 00:0e 6502       /usr/local/natinst/lib/libni_emb.so.13.0.0
ac190000-ac199000 r-xp 00000000 00:0e 4237       /usr/lib/arm-linux-gnueabi/libNiFpgaLv.so.22.5.0
ac199000-ac1a0000 ---p 00009000 00:0e 4237       /usr/lib/arm-linux-gnueabi/libNiFpgaLv.so.22.5.0
ac1a0000-ac1a1000 rw-p 00008000 00:0e 4237       /usr/lib/arm-linux-gnueabi/libNiFpgaLv.so.22.5.0
ac1a1000-ac1b2000 r-xp 00000000 00:0e 6488       /usr/local/natinst/lib/libnirio_emb_can.so.21.0.0
ac1b2000-ac1ba000 ---p 00011000 00:0e 6488       /usr/local/natinst/lib/libnirio_emb_can.so.21.0.0
ac1ba000-ac1bb000 rw-p 00011000 00:0e 6488       /usr/local/natinst/lib/libnirio_emb_can.so.21.0.0
ac1bb000-ac1ea000 r-xp 00000000 00:0e 4212       /usr/lib/arm-linux-gnueabi/libvisa.so.22.5.0
ac1ea000-ac1f2000 ---p 0002f000 00:0e 4212       /usr/lib/arm-linux-gnueabi/libvisa.so.22.5.0
ac1f2000-ac1f3000 rw-p 0002f000 00:0e 4212       /usr/lib/arm-linux-gnueabi/libvisa.so.22.5.0
ac1f3000-ac203000 rw-p 00000000 00:00 0
ac203000-ac2c3000 r-xp 00000000 00:0e 6416       /usr/local/frc/lib/libRoboRIO_FRC_ChipObject.so.23.0.0
ac2c3000-ac2ca000 ---p 000c0000 00:0e 6416       /usr/local/frc/lib/libRoboRIO_FRC_ChipObject.so.23.0.0
ac2ca000-ac2d0000 rw-p 000bf000 00:0e 6416       /usr/local/frc/lib/libRoboRIO_FRC_ChipObject.so.23.0.0
ac2d0000-ac38b000 r-xp 00000000 00:0e 6418       /usr/local/frc/lib/libFRC_NetworkCommunication.so.23.0.0
ac38b000-ac393000 ---p 000bb000 00:0e 6418       /usr/local/frc/lib/libFRC_NetworkCommunication.so.23.0.0
ac393000-ac396000 rw-p 000bb000 00:0e 6418       /usr/local/frc/lib/libFRC_NetworkCommunication.so.23.0.0
ac396000-ac39a000 r-xp 00000000 00:0e 4120       /usr/lib/libatomic.so.1.2.0
ac39a000-ac3a9000 ---p 00004000 00:0e 4120       /usr/lib/libatomic.so.1.2.0
ac3a9000-ac3aa000 r--p 00003000 00:0e 4120       /usr/lib/libatomic.so.1.2.0
ac3aa000-ac3ab000 rw-p 00004000 00:0e 4120       /usr/lib/libatomic.so.1.2.0
ac3ab000-ac3ac000 rw-p 00000000 00:00 0
ac3ac000-ac477000 r-xp 00000000 00:0e 11132      /usr/local/frc/third-party/lib/libwpiutil.so
ac477000-ac487000 ---p 000cb000 00:0e 11132      /usr/local/frc/third-party/lib/libwpiutil.so
ac487000-ac488000 r--p 000cb000 00:0e 11132      /usr/local/frc/third-party/lib/libwpiutil.so
ac488000-ac489000 rw-p 000cc000 00:0e 11132      /usr/local/frc/third-party/lib/libwpiutil.so
ac489000-ac48b000 rw-p 00000000 00:00 0
ac48b000-ac51b000 r-xp 00000000 00:0e 11135      /usr/local/frc/third-party/lib/libwpiHal.so
ac51b000-ac52a000 ---p 00090000 00:0e 11135      /usr/local/frc/third-party/lib/libwpiHal.so
ac52a000-ac52b000 r--p 0008f000 00:0e 11135      /usr/local/frc/third-party/lib/libwpiHal.so
ac52b000-ac52c000 rw-p 00090000 00:0e 11135      /usr/local/frc/third-party/lib/libwpiHal.so
ac52c000-ac531000 rw-p 00000000 00:00 0
ac531000-ac57e000 r-xp 00000000 00:0e 11131      /usr/local/frc/third-party/lib/libwpiHaljni.so
ac57e000-ac58e000 ---p 0004d000 00:0e 11131      /usr/local/frc/third-party/lib/libwpiHaljni.so
ac58e000-ac58f000 r--p 0004d000 00:0e 11131      /usr/local/frc/third-party/lib/libwpiHaljni.so
ac58f000-ac591000 rw-p 0004e000 00:0e 11131      /usr/local/frc/third-party/lib/libwpiHaljni.so
ac591000-ac592000 ---p 00000000 00:00 0
ac592000-ac612000 rw-p 00000000 00:00 0
ac612000-ac615000 ---p 00000000 00:00 0
ac615000-ac662000 rw-p 00000000 00:00 0
ac662000-ac665000 ---p 00000000 00:00 0
ac665000-ac6b2000 rw-p 00000000 00:00 0
ac6b2000-ac6b4000 r-xp 00000000 00:0e 11185      /usr/local/frc/JRE/lib/libextnet.so
ac6b4000-ac6c3000 ---p 00002000 00:0e 11185      /usr/local/frc/JRE/lib/libextnet.so
ac6c3000-ac6c4000 r--p 00001000 00:0e 11185      /usr/local/frc/JRE/lib/libextnet.so
ac6c4000-ac6c5000 rw-p 00002000 00:0e 11185      /usr/local/frc/JRE/lib/libextnet.so
ac6c5000-ac6ca000 r-xp 00000000 00:0e 2383       /lib/libnss_mdns.so.2
ac6ca000-ac6d9000 ---p 00005000 00:0e 2383       /lib/libnss_mdns.so.2
ac6d9000-ac6da000 r--p 00004000 00:0e 2383       /lib/libnss_mdns.so.2
ac6da000-ac6db000 rw-p 00005000 00:0e 2383       /lib/libnss_mdns.so.2
ac6db000-ac6ed000 r-xp 00000000 00:0e 2431       /lib/libresolv-2.24.so
ac6ed000-ac6fc000 ---p 00012000 00:0e 2431       /lib/libresolv-2.24.so
ac6fc000-ac6fd000 r--p 00011000 00:0e 2431       /lib/libresolv-2.24.so
ac6fd000-ac6fe000 rw-p 00012000 00:0e 2431       /lib/libresolv-2.24.so
ac6fe000-ac800000 rw-p 00000000 00:00 0
ac808000-ac80c000 r-xp 00000000 00:0e 2476       /lib/libnss_dns-2.24.so
ac80c000-ac81b000 ---p 00004000 00:0e 2476       /lib/libnss_dns-2.24.so
ac81b000-ac81c000 r--p 00003000 00:0e 2476       /lib/libnss_dns-2.24.so
ac81c000-ac81d000 rw-p 00004000 00:0e 2476       /lib/libnss_dns-2.24.so
ac81d000-ac82f000 r-xp 00000000 00:0e 11196      /usr/local/frc/JRE/lib/libzip.so
ac82f000-ac83e000 ---p 00012000 00:0e 11196      /usr/local/frc/JRE/lib/libzip.so
ac83e000-ac83f000 r--p 00011000 00:0e 11196      /usr/local/frc/JRE/lib/libzip.so
ac83f000-ac840000 rw-p 00012000 00:0e 11196      /usr/local/frc/JRE/lib/libzip.so
ac840000-ac84f000 r-xp 00000000 00:0e 11192      /usr/local/frc/JRE/lib/libnet.so
ac84f000-ac85e000 ---p 0000f000 00:0e 11192      /usr/local/frc/JRE/lib/libnet.so
ac85e000-ac85f000 r--p 0000e000 00:0e 11192      /usr/local/frc/JRE/lib/libnet.so
ac85f000-ac860000 rw-p 0000f000 00:0e 11192      /usr/local/frc/JRE/lib/libnet.so
ac860000-ac863000 ---p 00000000 00:00 0
ac863000-ac8b0000 rw-p 00000000 00:00 0
ac8b0000-ac8b3000 ---p 00000000 00:00 0
ac8b3000-ac9fd000 rw-p 00000000 00:00 0
ac9fd000-aca00000 ---p 00000000 00:00 0
aca01000-aca09000 r-xp 00000000 00:0e 4235       /usr/lib/arm-linux-gnueabi/libniriosession.so.22.5.0
aca09000-aca0a000 rw-p 00008000 00:0e 4235       /usr/lib/arm-linux-gnueabi/libniriosession.so.22.5.0
aca0a000-aca0e000 r-xp 00000000 00:0e 2447       /lib/libnss_mdns4_minimal.so.2
aca0e000-aca1d000 ---p 00004000 00:0e 2447       /lib/libnss_mdns4_minimal.so.2
aca1d000-aca1e000 r--p 00003000 00:0e 2447       /lib/libnss_mdns4_minimal.so.2
aca1e000-aca1f000 rw-p 00004000 00:0e 2447       /lib/libnss_mdns4_minimal.so.2
aca1f000-acc00000 r--p 00000000 00:0e 3933       /usr/lib/locale/locale-archive
acc00000-ad000000 rw-p 00000000 00:00 0
...
ad380000-ad400000 rw-p 00000000 00:00 0
ad405000-ad40a000 r-xp 00000000 00:0e 11172      /usr/local/frc/JRE/lib/libmanagement_ext.so
ad40a000-ad419000 ---p 00005000 00:0e 11172      /usr/local/frc/JRE/lib/libmanagement_ext.so
ad419000-ad41a000 r--p 00004000 00:0e 11172      /usr/local/frc/JRE/lib/libmanagement_ext.so
ad41a000-ad41b000 rw-p 00005000 00:0e 11172      /usr/local/frc/JRE/lib/libmanagement_ext.so
ad41b000-ad41e000 ---p 00000000 00:00 0
ad41e000-ad46b000 rw-p 00000000 00:00 0
...
ad880000-ad95a000 rw-p 00000000 00:00 0
ad95a000-ada00000 ---p 00000000 00:00 0
ada06000-ada13000 r-xp 00000000 00:0e 11161      /usr/local/frc/JRE/lib/libnio.so
ada13000-ada22000 ---p 0000d000 00:0e 11161      /usr/local/frc/JRE/lib/libnio.so
ada22000-ada23000 r--p 0000c000 00:0e 11161      /usr/local/frc/JRE/lib/libnio.so
ada23000-ada24000 rw-p 0000d000 00:0e 11161      /usr/local/frc/JRE/lib/libnio.so
ada24000-ada7f000 rw-p 00000000 00:00 0
ada7f000-ada80000 ---p 00000000 00:00 0
ada80000-adbfc000 rw-p 00000000 00:00 0
adbfc000-adc00000 ---p 00000000 00:00 0
adc04000-adc07000 r-xp 00000000 00:0e 11165      /usr/local/frc/JRE/lib/libmanagement.so
adc07000-adc16000 ---p 00003000 00:0e 11165      /usr/local/frc/JRE/lib/libmanagement.so
adc16000-adc17000 r--p 00002000 00:0e 11165      /usr/local/frc/JRE/lib/libmanagement.so
adc17000-adc18000 rw-p 00003000 00:0e 11165      /usr/local/frc/JRE/lib/libmanagement.so
adc18000-adc4f000 r--p 00000000 00:0e 3927       /usr/lib/locale/L1/LC_CTYPE
adc4f000-adc50000 ---p 00000000 00:00 0
...
aef00000-b4000000 ---p 00000000 00:00 0
b4004000-b4006000 r-xp 00000000 00:0e 6483       /usr/local/natinst/lib/libni_rtlog.so.2.10.0
b4006000-b400e000 ---p 00002000 00:0e 6483       /usr/local/natinst/lib/libni_rtlog.so.2.10.0
b400e000-b400f000 rw-p 00002000 00:0e 6483       /usr/local/natinst/lib/libni_rtlog.so.2.10.0
b400f000-b4037000 rw-p 00000000 00:00 0
...
b435f000-b610f000 ---p 00000000 00:00 0
b610f000-b6126000 r-xp 00000000 00:0e 11166      /usr/local/frc/JRE/lib/libjava.so
b6126000-b6136000 ---p 00017000 00:0e 11166      /usr/local/frc/JRE/lib/libjava.so
b6136000-b6137000 r--p 00017000 00:0e 11166      /usr/local/frc/JRE/lib/libjava.so
b6137000-b6138000 rw-p 00018000 00:0e 11166      /usr/local/frc/JRE/lib/libjava.so
b6138000-b62c7000 r-xp 00000000 00:0e 2390       /lib/libnss_niauth.so.21.5.0
b62c7000-b62ce000 ---p 0018f000 00:0e 2390       /lib/libnss_niauth.so.21.5.0
b62ce000-b62ec000 rw-p 0018e000 00:0e 2390       /lib/libnss_niauth.so.21.5.0
b62ec000-b62ef000 rw-p 00000000 00:00 0
b62ef000-b6400000 r--s 00000000 00:0e 11186      /usr/local/frc/JRE/lib/modules
b6400000-b64fb000 rw-p 00000000 00:00 0
b64fb000-b6500000 ---p 00000000 00:00 0
b650b000-b650c000 rw-s 00065000 00:06 10049      /dev/ni/niembcanzynqk:0\niembcan_ucan\0
b650c000-b650d000 r--p 00000000 00:0e 3932       /usr/lib/locale/L1/LC_NUMERIC
b650d000-b650e000 r--p 00000000 00:0e 3925       /usr/lib/locale/L1/LC_TIME
b650e000-b6513000 r--p 00000000 00:0e 3921       /usr/lib/locale/L1/LC_COLLATE
b6513000-b6514000 r--p 00000000 00:0e 3930       /usr/lib/locale/L1/LC_MONETARY
b6514000-b6515000 r--p 00000000 00:0e 3923       /usr/lib/locale/L1/LC_MESSAGES/SYS_LC_MESSAGES
b6515000-b651e000 r-xp 00000000 00:0e 2435       /lib/libnss_files-2.24.so
b651e000-b652d000 ---p 00009000 00:0e 2435       /lib/libnss_files-2.24.so
b652d000-b652e000 r--p 00008000 00:0e 2435       /lib/libnss_files-2.24.so
b652e000-b652f000 rw-p 00009000 00:0e 2435       /lib/libnss_files-2.24.so
b652f000-b6535000 rw-p 00000000 00:00 0
b6535000-b653e000 r-xp 00000000 00:0e 2398       /lib/libnss_nis-2.24.so
b653e000-b654d000 ---p 00009000 00:0e 2398       /lib/libnss_nis-2.24.so
b654d000-b654e000 r--p 00008000 00:0e 2398       /lib/libnss_nis-2.24.so
b654e000-b654f000 rw-p 00009000 00:0e 2398       /lib/libnss_nis-2.24.so
b654f000-b6560000 r-xp 00000000 00:0e 2462       /lib/libnsl-2.24.so
b6560000-b656f000 ---p 00011000 00:0e 2462       /lib/libnsl-2.24.so
b656f000-b6570000 r--p 00010000 00:0e 2462       /lib/libnsl-2.24.so
b6570000-b6571000 rw-p 00011000 00:0e 2462       /lib/libnsl-2.24.so
b6571000-b6573000 rw-p 00000000 00:00 0
b6573000-b6579000 r-xp 00000000 00:0e 2438       /lib/libnss_compat-2.24.so
b6579000-b6589000 ---p 00006000 00:0e 2438       /lib/libnss_compat-2.24.so
b6589000-b658a000 r--p 00006000 00:0e 2438       /lib/libnss_compat-2.24.so
b658a000-b658b000 rw-p 00007000 00:0e 2438       /lib/libnss_compat-2.24.so
b658b000-b6593000 rw-s 00000000 00:11 60905      /var/volatile/tmp/hsperfdata_lvuser/1588
b6593000-b6596000 r-xp 00000000 00:0e 11183      /usr/local/frc/JRE/lib/libjimage.so
b6596000-b65a5000 ---p 00003000 00:0e 11183      /usr/local/frc/JRE/lib/libjimage.so
b65a5000-b65a6000 r--p 00002000 00:0e 11183      /usr/local/frc/JRE/lib/libjimage.so
b65a6000-b65a7000 rw-p 00003000 00:0e 11183      /usr/local/frc/JRE/lib/libjimage.so
b65a7000-b65aa000 ---p 00000000 00:00 0
b65aa000-b65f7000 rw-p 00000000 00:00 0
b65f7000-b6612000 r-xp 00000000 00:0e 2452       /lib/libgcc_s.so.1
b6612000-b6621000 ---p 0001b000 00:0e 2452       /lib/libgcc_s.so.1
b6621000-b6622000 r--p 0001a000 00:0e 2452       /lib/libgcc_s.so.1
b6622000-b6623000 rw-p 0001b000 00:0e 2452       /lib/libgcc_s.so.1
b6623000-b6690000 r-xp 00000000 00:0e 2482       /lib/libm-2.24.so
b6690000-b669f000 ---p 0006d000 00:0e 2482       /lib/libm-2.24.so
b669f000-b66a0000 r--p 0006c000 00:0e 2482       /lib/libm-2.24.so
b66a0000-b66a1000 rw-p 0006d000 00:0e 2482       /lib/libm-2.24.so
b66a1000-b6850000 r-xp 00000000 00:0e 3948       /usr/lib/libstdc++.so.6.0.30
b6850000-b685f000 ---p 001af000 00:0e 3948       /usr/lib/libstdc++.so.6.0.30
b685f000-b6864000 r--p 001ae000 00:0e 3948       /usr/lib/libstdc++.so.6.0.30
b6864000-b6866000 rw-p 001b3000 00:0e 3948       /usr/lib/libstdc++.so.6.0.30
b6866000-b6868000 rw-p 00000000 00:00 0
b6868000-b686e000 r-xp 00000000 00:0e 2440       /lib/librt-2.24.so
b686e000-b687d000 ---p 00006000 00:0e 2440       /lib/librt-2.24.so
b687d000-b687e000 r--p 00005000 00:0e 2440       /lib/librt-2.24.so
b687e000-b687f000 rw-p 00006000 00:0e 2440       /lib/librt-2.24.so
b687f000-b6dbd000 r-xp 00000000 00:0e 11189      /usr/local/frc/JRE/lib/client/libjvm.so
b6dbd000-b6dcc000 ---p 0053e000 00:0e 11189      /usr/local/frc/JRE/lib/client/libjvm.so
b6dcc000-b6de3000 r--p 0053d000 00:0e 11189      /usr/local/frc/JRE/lib/client/libjvm.so
b6de3000-b6df3000 rw-p 00554000 00:0e 11189      /usr/local/frc/JRE/lib/client/libjvm.so
b6df3000-b6e19000 rw-p 00000000 00:00 0
b6e19000-b6f3e000 r-xp 00000000 00:0e 3085       /lib/libc-2.24.so
b6f3e000-b6f4e000 ---p 00125000 00:0e 3085       /lib/libc-2.24.so
b6f4e000-b6f50000 r--p 00125000 00:0e 3085       /lib/libc-2.24.so
b6f50000-b6f51000 rw-p 00127000 00:0e 3085       /lib/libc-2.24.so
b6f51000-b6f54000 rw-p 00000000 00:00 0
b6f54000-b6f56000 r-xp 00000000 00:0e 2455       /lib/libdl-2.24.so
b6f56000-b6f65000 ---p 00002000 00:0e 2455       /lib/libdl-2.24.so
b6f65000-b6f66000 r--p 00001000 00:0e 2455       /lib/libdl-2.24.so
b6f66000-b6f67000 rw-p 00002000 00:0e 2455       /lib/libdl-2.24.so
b6f67000-b6f7d000 r-xp 00000000 00:0e 2392       /lib/libpthread-2.24.so
b6f7d000-b6f8c000 ---p 00016000 00:0e 2392       /lib/libpthread-2.24.so
b6f8c000-b6f8d000 r--p 00015000 00:0e 2392       /lib/libpthread-2.24.so
b6f8d000-b6f8e000 rw-p 00016000 00:0e 2392       /lib/libpthread-2.24.so
b6f8e000-b6f90000 rw-p 00000000 00:00 0
b6f90000-b6fa1000 r-xp 00000000 00:0e 11177      /usr/local/frc/JRE/lib/libjli.so
b6fa1000-b6fb0000 ---p 00011000 00:0e 11177      /usr/local/frc/JRE/lib/libjli.so
b6fb0000-b6fb1000 r--p 00010000 00:0e 11177      /usr/local/frc/JRE/lib/libjli.so
b6fb1000-b6fb2000 rw-p 00011000 00:0e 11177      /usr/local/frc/JRE/lib/libjli.so
b6fb2000-b6fd2000 r-xp 00000000 00:0e 2389       /lib/ld-2.24.so
b6fd2000-b6fd3000 r--p 00000000 00:0e 3928       /usr/lib/locale/L1/LC_PAPER
b6fd3000-b6fd4000 r--p 00000000 00:0e 3924       /usr/lib/locale/L1/LC_NAME
b6fd4000-b6fd5000 r--p 00000000 00:0e 3926       /usr/lib/locale/L1/LC_ADDRESS
b6fd5000-b6fd7000 rw-p 00000000 00:00 0
b6fd7000-b6fd8000 r--p 00000000 00:0e 3929       /usr/lib/locale/L1/LC_TELEPHONE
b6fd8000-b6fd9000 r--p 00000000 00:0e 3931       /usr/lib/locale/L1/LC_MEASUREMENT
b6fd9000-b6fda000 r--p 00000000 00:0e 3920       /usr/lib/locale/L1/LC_IDENTIFICATION
b6fda000-b6fdc000 rw-p 00000000 00:00 0
...
b6fdf000-b6fe1000 rw-p 00000000 00:00 0
b6fe1000-b6fe2000 r--p 0001f000 00:0e 2389       /lib/ld-2.24.so
b6fe2000-b6fe3000 rw-p 00020000 00:0e 2389       /lib/ld-2.24.so
bed0a000-bed2b000 rw-p 00000000 00:00 0          [stack]
bed79000-bed7a000 r-xp 00000000 00:00 0          [sigpage]
ffff0000-ffff1000 r-xp 00000000 00:00 0          [vectors]

Peter_Johnson · August 7, 2023, 2:42am

No, you can’t just remove opencv. It’s pulled in by cscore, which is pulled in by CameraServer, which is pulled in by RobotBase. It shouldn’t be doing any harm, it’s just mmap’ed in, and unless it’s actually being used it shouldn’t even be pulled into physical memory.

We know the GC settings need additional tuning. You could try tweaking the GC settings or downgrading to Java 11 (see GitHub - wpilibsuite/GradleRIO: The official gradle plugin for the FIRST Robotics Competition) but we will likely remove the downgrade option sometime in the future so we can start using Java 17 features in the library.

truher · August 7, 2023, 2:58am

heh, ok, thanks.

re: gc tuning, i did try that a little (e.g. the serial one), and it did behave differently but it didn’t really cure the memory errors.

what i really want to know is why my basic measurements seem wrong, 70MB RSS on a machine with 100MB free and it OOMs. is there any way to look more closely into that? i feel like i’m still far away from seeing the actual problem. can i run valgrind on the roborio?

Peter_Johnson · August 7, 2023, 3:29am

Not sure if /proc/buddyinfo is available on the Rio (looking at the first answer to Linux oom situation (32 bit kernel) - Server Fault), but may be worth looking at to see if there’s a lot of allocation fragmentation happening.

It should be possible to run valgrind (valgrind says armv7 is a supported platform), but you’ll have to compile it yourself and I’m somewhat skeptical it will work sufficiently to give you useful information.

truher · August 7, 2023, 4:59am

yeah, buddyinfo exists.

here it is in an unhappy state:

admin@roboRIO-100-FRC:~# cat /proc/buddyinfo
Node 0, zone   Normal      1      2      8      6      1      2      0      4      3      2      3

here it is in a good state:

admin@roboRIO-100-FRC:~# cat /proc/buddyinfo
Node 0, zone   Normal    187    231    161     47      8      7      0      3      3      2      3

The difference here whether the NI webserver is running. when it is, the robot code can run for a minute or so before OOMing on mallocs ranging from 32KB to 1MB, though there are 3 1MB pages available. with the webserver off, the robot code runs indefinitely. the total ram here seems about the same as the “MemFree” number, not the “MemAvailable” number, which i guess makes sense.

i switched to the serial gc, with the webserver running, and the jmx server serving (it often crashes the robot code) and got the surprising result that the robot stayed up indefinitely, but i also got more fragmentation, which might just be caused by everything running longer.

admin@roboRIO-100-FRC:/tmp# cat /proc/buddyinfo
Node 0, zone   Normal     63    107     83     21      3      0      1      2      0      0      0

the serial gc pauses were reported (“Pause Young (Allocation Failure)”) as about 5ms, and G1 produced about 12ms pauses, which is kinda hard to believe actually.

btw, is there a normal way to make sure the NI webserver stays off? i could remove all the rc files manually but maybe gradle could do it somehow?

calcmogul · August 7, 2023, 5:08am

My former team used these scripts to disable the service, and enable it again if we needed to reimage the roboRIO.

github.com

frc3512/Robot-2020/blob/main/tools/start_system_web_server.py

#!/usr/bin/env python3

import subprocess

import frcutils

subprocess.run([
    "ssh", f"admin@{frcutils.get_roborio_ip()}",
    "/usr/local/natinst/etc/init.d/systemWebServer", "start"
])

github.com

frc3512/Robot-2020/blob/main/tools/stop_system_web_server.py

#!/usr/bin/env python3

import subprocess

import frcutils

subprocess.run([
    "ssh", f"admin@{frcutils.get_roborio_ip()}",
    "/usr/local/natinst/etc/init.d/systemWebServer", "stop"
])

The memory leaks would eventually cause our roboRIO to lock up (C++ RT robot program).

truher · August 7, 2023, 6:47pm

Thank you!

FYI for the curious:

As an experiment, I set up a USB flash drive as swap space, and to my surprise it worked fine, it swaps out about 10MB of stuff when the jvm starts and then it leaves it alone. it seems like the JVM needs quite a large heap to start (i.e. it fails if Xmx is below 100MB) but when running it only needs ~10MB, and gives a lot of the difference back to the kernel, so maybe if any of the swapped-out stuff needs to actually run (e.g. FRC_NetCommDaemon is one of them), it can happily return.

another experiment with kernel parameters (/etc/sysctl.conf)

swappiness=100 (use as much swap as possible, default is 60)
vfs_cache_pressure=1000 (avoid caching files, default is 100)
overcommit_memory=1 (always overcommit, default is 2=never)
also removed -XX:_AlwaysPreTouch in the jvm

this also worked fine, without using any swap, and also without changing the cache level much, maybe because the jvm asks for a lot of ram without using it, so there isn’t really any cache pressure or swap pressure when overcommit is unlimited.

the whole time the “available” ram remains 100MB

truher · August 8, 2023, 2:34am

one more update, it turns out just turning off “pre-touching” works by itself, without any of the kernel parameter changes, which i can rationalize by thinking about pre-touching the ocean of actually-never-used heap that the greedy jvm allocates for itself.

also if you want to muck with the rio config, you can do it with gradlerio. we made an “etc” directory next to the “deploy” directory, so add these to the “artifacts” list under the “deploy” one:

    frcStaticFileDeploy(getArtifactTypeClass('FileTreeArtifact')) {
        files = project.fileTree('src/main/etc')
        directory = '/etc'
    }
    reloadSysctl(getArtifactTypeClass('CommandArtifact')) {
        command = 'sysctl -p'
    }

truher · August 8, 2023, 2:47am

ha ha ok, i was wrong, i had turned off the webserver during the no-pre-touching-only test.

with the webserver on, you need both jvm no-pre-touching and kernel infinite-over-committing, but you don’t need swap or hyper-cache-pressure.

then it works fine.