Roborio Using Too much RAM

Recently, our team (Galaxia in memory of David Zohar) ran into a puzzling problem. It appears that the Roborio is using too much RAM for the robot code. After some poking around in the Roborio using an SSH, we ran a top command, and this came up:

As you can see at the top of the top (pun intended), the most prominent process is some kind of java process. We tried deleting the robot code jar file from the Roborio while the code is running, and saw that the process disappeared, meaning that it is in fact the robot code. Ultimately, this means that the robot code is using 209 MB of RAM. At this point we decided to an empty project to see if the problem lies in the size of our robot code (or some kind of large loop), but the RAM usage only reduced by 7 MB. We tried changing the Roborio, and the same exact problem came up (both with the robot code and the empty project). Finally, we tried to change the computer that we used to run these tests, and still the problem came up.

After some research, and opening a topic in three different discord servers, we came across the following solution, as provided by WPILib:
Add this line of code to the build.gradle (doesn’t matter where):
wpi.jreArtifactLocation = 'edu.wpi.first.jdk:roborio-2021:11.0.9u11-1'
And add this line of code to the build.gradle in the frcJava block in the artifacts:
gcType = 'CMS'
This solution suggested that the garbage collector that the roborio is running isn’t very effective, and hence it is necessary to revert to an earlier version.
Unfortunately, this had no effect on the performance of the Roborio.

Additionally, we attempted to add the parameter -XX:MaxDirectMemorySize to the jvmArgs right after the line specifying the gc type (as shown above). At first this appeared to fix the problem, but after some more testing the problem came back. We were unable to recreate the conditions that might have caused it to work.


(The 0mb value of the parameter was just to check that it had an effect)

We haven’t come across anybody that has run into this problem, but some have upvoted our post on discord (not sure what that means), and so we’re wondering if anybody else came across this problem and/or knows how to solve it.

Thank you in advance for your help.

7 Likes

Have you actually run out of memory, are are you just concerned about the amount shown. See Unusually high memory usage by WPILib - #3 by Peter_Johnson for why top isn’t a good indication of memory usage.

2 Likes

Ah, I forgot to mention. Half of the time this doesn’t cause a problem, and the other half it causes a memory allocation error. Also, we’ve read that thread, and we started looking at the DriverStation telemetry. This showed that the Roborio had 4 MB left.

1 Like

It’s possible you’re leaking memory somewhere. My best guess would be calling new on something every cycle–search your code for that.

1 Like

Like I said in the original post, an empty project causes the same problem. This means that if there is a memory leak, it’s in the wpilib libraries.

1 Like

How did you create the empty project?

2 Likes

Using vscode.

1 Like

Which template (So we can try to reproduce the same thing)?

Please run these commands (also given in my post linked by Joe above) and post the results here. VSZ in top and “free memory” in top are not indicative of actual memory usage.

cat /proc/meminfo
cat /proc/PID/status

where PID=the PID of the FRCUserProgram as shown in top.

The numbers we care about (eg are actually indicative of the memory state) are MemAvailable in the first command and VmHWM and VmRSS in the second command.

1 Like

And run it both on your robot code and the vs code template project.

Command based robot.

1 Like

We’re not working on this issue right now, but we’ll try it.

Ok, so we ran it on the robot code and this is what came out:
cat proc/meminfo -
MemTotal: 250152 kB MemFree: 3444 kB MemAvailable: 112300 kB Buffers: 0 kB Cached: 130032 kB SwapCached: 0 kB Active: 126368 kB Inactive: 44248 kB Active(anon): 63732 kB Inactive(anon): 664 kB Active(file): 62636 kB Inactive(file): 43584 kB Unevictable: 35884 kB Mlocked: 35884 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 250152 kB LowFree: 3444 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 76488 kB Mapped: 59804 kB Shmem: 836 kB Slab: 20136 kB SReclaimable: 11120 kB SUnreclaim: 9016 kB KernelStack: 2208 kB PageTables: 1748 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 215128 kB Committed_AS: 159616 kB VmallocTotal: 770048 kB VmallocUsed: 0 kB VmallocChunk: 0 kB
cat proc/PID/status -
Name: java Umask: 0022 State: S (sleeping) Tgid: 2817 Ngid: 0 Pid: 2817 PPid: 2806 TracerPid: 0 Uid: 500 500 500 500 Gid: 500 500 500 500 FDSize: 256 Groups: 5 44 46 397 498 500 VmPeak: 2967152 kB VmSize: 223384 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 55404 kB VmRSS: 55404 kB RssAnon: 34364 kB RssFile: 20496 kB RssShmem: 544 kB VmData: 49928 kB VmStk: 132 kB VmExe: 4 kB VmLib: 34432 kB VmPTE: 110 kB VmPMD: 0 kB VmSwap: 0 kB Threads: 48 SigQ: 0/1946 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000003000 SigCgt: 2000000181004ccf CapInh: 0000000000000000 CapPrm: 0000000000800000 CapEff: 0000000000800000 CapBnd: 0000003fffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Speculation_Store_Bypass: unknown Cpus_allowed: 3 Cpus_allowed_list: 0-1 Mems_allowed: 1 Mems_allowed_list: 0 voluntary_ctxt_switches: 1 nonvoluntary_ctxt_switches: 11

After trying the template code, the driver station writes that there is 33 MB of free RAM. This completely differs to the previous 10 MB that it showed.
image
Here are the results of the commands:
cat /proc/3044/status-
Name: java Umask: 0022 State: S (sleeping) Tgid: 3044 Ngid: 0 Pid: 3044 PPid: 3033 TracerPid: 0 Uid: 500 500 500 500 Gid: 500 500 500 500 FDSize: 256 Groups: 5 44 46 397 498 500 VmPeak: 2967152 kB VmSize: 214140 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 43532 kB VmRSS: 43532 kB RssAnon: 24944 kB RssFile: 18544 kB RssShmem: 44 kB VmData: 36936 kB VmStk: 132 kB VmExe: 4 kB VmLib: 31896 kB VmPTE: 92 kB VmPMD: 0 kB VmSwap: 0 kB Threads: 36 SigQ: 0/1946 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000003000 SigCgt: 2000000181004ccf CapInh: 0000000000000000 CapPrm: 0000000000800000 CapEff: 0000000000800000 CapBnd: 0000003fffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Speculation_Store_Bypass: unknown Cpus_allowed: 3 Cpus_allowed_list: 0-1 Mems_allowed: 1 Mems_allowed_list: 0 voluntary_ctxt_switches: 2 nonvoluntary_ctxt_switches: 16
cat /proc/meminfo-
MemTotal: 250152 kB MemFree: 31244 kB MemAvailable: 123560 kB Buffers: 0 kB Cached: 113972 kB SwapCached: 0 kB Active: 110684 kB Inactive: 33552 kB Active(anon): 53484 kB Inactive(anon): 656 kB Active(file): 57200 kB Inactive(file): 32896 kB Unevictable: 35628 kB Mlocked: 35628 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 250152 kB LowFree: 31244 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 65916 kB Mapped: 57360 kB Shmem: 900 kB Slab: 18928 kB SReclaimable: 10704 kB SUnreclaim: 8224 kB KernelStack: 2112 kB PageTables: 1720 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 215128 kB Committed_AS: 146324 kB VmallocTotal: 770048 kB VmallocUsed: 0 kB VmallocChunk: 0 kB

1 Like

The robot code is now stable at 15 MB!!!
image
I think this is a miracle.
However, I am curious as to why there was a sudden change, because I changed absolutely nothing.

Nevermind, it came right back.

These show that there is 112 and 123 mb of ram available to the system. “Free” memory (the smaller number that you are focusing on) is unused (read: wasted).

The available memory will be reclaimed if another process needs it, but the OS will leave it as is until that point.

This is the number that actually matters- the program is using 55/43MB of ram.

What specific error message were you getting? That would help us narrow down a root cause.

1 Like

Unless your noticing crashes or slow downs, i wouldn’t worry about this. Java often times allocates more memory than it uses, which sounds bad, but really isn’t.

There’s a difference between virtual memory and physical memory, which, TLDR, means that while java “allocated” all of your memory, it’s only using like half of it, linux (and really any good operating system) makes sure to map memory optimally.

It hasn’t been happening much recently, but when it does I’ll be sure to get you a screenshot of the error. If the system really does have that much free RAM, then how come the DriverStation shows that it doesn’t?

1 Like

I consider that to be a bug. It should report available memory, which is a much more useful metric. It has always reported “free” memory, but a few years ago a kernel upgrade resulted in the kernel more aggressively caching, which reduces “free” memory, and making that number less useful.

1 Like